Introduction

Omni makes keeping your cluster up-to-date easy - which is good, as it is important to stay current with Talos Linux and Kubernetes releases, to ensure you are not exposed to already fixed security issues and bugs. Keeping your clusters up-to-date involves updating both the underlying operating system (Talos Linux) and Kubernetes.

Upgrading the Operating System

In order to update the Talos Linux version of all nodes in a cluster, navigate to the overview of the cluster you wish to update. (For example, click the cluster name in the Clusters panel.) If newer Talos Linux versions are available, there will be an indication in the far right, where the current cluster Talos version is listed. Clicking that icon, or the "Update Talos" button in the lower right, will allow you to select the new version of Talos Linux that should be deployed across all nodes of the cluster.

Select the new version, and then "Upgrade" (or "Downgrade", if you are selecting an older version than currently deployed.) (Omni will ensure that the Kubernetes version running in the cluster is compatible with the selected version of Talos Linux.)

Note: You may upgrade to any Talos version shown. Omni will only allow supported upgrade paths. In some cases this may require an intermediate upgrade before upgrading to the most recent version.

Omni will then cycle through all nodes in the cluster, safely updating them to the selected version of Talos Linux. Omni will update the control plane nodes first. (Omni ensures the etcd cluster is healthy and will remain healthy after the node being updated leaves the etcd cluster, before allowing a control plane node to be upgraded.)

Omni will drain and cordon each node, update the OS, and then un-cordon the node. Omni always updates nodes with the Talos Linux flag --preserve=true, keeping ephemeral data.

NOTE: If any of your workloads are sensitive to being shut down ungracefully, be sure to use the lifecycle.preStop Pod spec.

Kubernetes Upgrades

As with the Talos Linux version, Omni will notify you on the right hand side of the cluster overview if there is a new version of Kubernetes available. You may click either the Upgrade icon next to the Kubernetes version, or the Update Kubernetes button on the lower right of the cluster overview. Kubernetes upgrades are done non-disruptively to workloads and are run in several phases:

• Images for new Kubernetes components are pre-pulled to the nodes to minimize downtime and test for image availability.

• New static pod definitions are rendered on the configuration update which is picked up by the kubelet. The command waits for the change to propagate to the API server state.

• The command updates the kube-proxy daemonset with the new image version.

• On every node in the cluster, the kubelet version is updated.

Note: The upgrade operation never deletes any resources from the cluster: obsolete resources should be deleted manually.

Applying changed Kubernetes Manifests

Unlike the Talos Linux command talosctl upgrade-k8s, Omni does not automatically apply updates to Kubernetes bootstrap manifests on a Kubernetes upgrade. This is to prevent Omni overwriting changes to the bootstrap manifests that you applied manually. (Talos Linux has a --dry-run feature on the upgrade command that shows you changes before the upgrade - Omni shows you the changes after the upgrade, but before they are applied.) Thus after each Kubernetes upgrade, it is recommended to examine the BootStrap Manifests of the cluster (as shown in the left hand navigation) and apply the changes, if they are appropriate.

Locking nodes

Omni allows you to control which nodes are upgraded during Talos or Kubernetes upgrade operations. You can lock nodes, which prevents them from receiving configuration updates, upgrades and downgrades. This allows you to ensure that new versions of Talos Linux or Kubernetes, or new config patches, are rolled out in a safe and controlled manner. If you cannot do a blue/green deployment with different clusters, you can roll out a new Kubernetes or Talos Linux release, or config patch, to just some of the nodes in your cluster. Once you have validated your applications perform correctly on the new versions, you can unlock all the nodes, and allow them to be updated also.

Note: you cannot lock control plane nodes, as it is not supported to have the Kubernetes version of a worker higher than that of the control plane nodes in a cluster - this may result in API version incompatibility.

To lock a node, simply select the Lock icon to the right of the node on the Cluster Overview screen, or use the omnictl cluster machine lock command. Upgrade and config patch operations will apply to all other nodes in the cluster, but locked nodes will retain their configuration at the time of locking. Unlock the nodes to allow pending cluster updates to complete.

Prerequisites

DNS server NTP server TLS certificates Installed on machine running Omni

• genuuid

  • Used to generate a unique account ID for Omni.

• Docker

  • Used for running the suite of applications

• Wireguard

  • Used by Siderolink

Overview

Gathering Dependencies

In this package, we will be installing:

• Gitea

• Keycloak

• Omni

To keep everything organized, I am using the following directory structure to store all the dependencies and I will move them to the airgapped network all at once.

NOTE: The empty directories will be used for the persistent data volumes when we deploy these apps in Docker.

airgap
├── certs
├── gitea
├── keycloak
├── omni
└── registry

Generate Certificates

TLS Certificates

Omni Certificate

Omni uses etcd to store the data for our installation and we need to give it a private key to use for encryption of the etcd database.

1. First, Generate a GPG key.

gpg --quick-generate-key "Omni (Used for etcd data encryption) how-to-guide@siderolabs.com" rsa4096 cert never

This will generate a new GPG key pair with the specified properties.

What's going on here?

• quick-generate-key allows us to quickly generate a new GPG key pair. -"Omni (Used for etcd data encryption) how-to-guide@siderolabs.com" is the user ID associated with the key which generally consists of the real name, a comment, and an email address for the user.

• rsa4096 specifies the algorithm type and key size.

• cert means this key can be used to certify other keys.

• never specifies that this key will never expire.

1. Add an encryption subkey

We will use the fingerprint of this key to create an encryption subkey.

To find the fingerprint of the key we just created, run:

gpg --list-secret-keys

Next, run the following command to create the encryption subkey, replacing $FPR with your own keys fingerprint.

gpg --quick-add-key $FPR rsa4096 encr never

In this command:

• $FPR is the fingerprint of the key we are adding the subkey to.

• rsa4096 and encr specify that the new subkey will be an RSA encryption key with a size of 4096 bits.

• never means this subkey will never expire.

1. Export the secret key

Lastly we'll export this key into an ASCII formatted file so Omni can use it.

gpg --export-secret-key --armor how-to-guide@siderolabs.com > certs/omni.asc

• --armor is an option which creates the output in ASCII format. Without it, the output would be binary.

Save this file to the certs directory in our package.

Create the app.ini File

Gitea uses a configuration file named app.ini which we can use to pre-configure with the necessary information to run Gitea and bypass the intitial startup page. When we start the container, we will mount this file as a volume using Docker.

Create the app.ini file

vim gitea/app.ini

Replace the DOMAIN, SSH_DOMAIN, and ROOT_URL values with your own hostname:

APP_NAME=Gitea: Git with a cup of tea
RUN_MODE=prod
RUN_USER=git
I_AM_BEING_UNSAFE_RUNNING_AS_ROOT=false

[server]
CERT_FILE=cert.pem
KEY_FILE=key.pem
APP_DATA_PATH=/data/gitea
DOMAIN=${GITEA_HOSTNAME}
SSH_DOMAIN=${GITEA_HOSTNAME}
HTTP_PORT=3000
ROOT_URL=https://${GITEA_HOSTNAME}:3000/
HTTP_ADDR=0.0.0.0
PROTOCOL=https
LOCAL_ROOT_URL=https://localhost:3000/

[database]
PATH=/data/gitea/gitea.db
DB_TYPE=sqlite3
HOST=localhost:3306
NAME=gitea
USER=root
PASSWD=

[security]
INSTALL_LOCK=true # This is the value which tells Gitea not to run the intitial configuration wizard on start up

NOTE: If running this in a production environment, you will also want to configure the database settings for a production database. This configuration will use an internal sqlite database in the container.

Gathering Images

Next we will gather all the images needed installing Gitea, Keycloak, Omni, and the images Omni will need for creating and installing Talos.

I'll be using the following images for the tutorial:

Gitea

• docker.io/gitea/gitea:1.19.3 Keycloak

• quay.io/keycloak/keycloak:21.1.1 Omni

• ghcr.io/siderolabs/omni:v0.31.0

• ghcr.io/siderolabs/imager:v1.4.5

  • pull this image to match the version of Talos you would like to use. Talos

• ghcr.io/siderolabs/flannel:v0.21.4

• ghcr.io/siderolabs/install-cni:v1.4.0-1-g9b07505

• docker.io/coredns/coredns:1.10.1

• gcr.io/etcd-development/etcd:v3.5.9

• registry.k8s.io/kube-apiserver:v1.27.2

• registry.k8s.io/kube-controller-manager:v1.27.2

• registry.k8s.io/kube-scheduler:v1.27.2

• registry.k8s.io/kube-proxy:v1.27.2

• ghcr.io/siderolabs/kubelet:v1.27.2

• ghcr.io/siderolabs/installer:v1.4.5

• registry.k8s.io/pause:3.6

Package the images

1. Pull the images to load them locally into Docker.

• Run the following command for each of the images listed above except for the Omni image which will be provided to you as an archive file already.

sudo docker pull registry/repository/image-name:tag

1. Verify all of the images have been downloaded

sudo docker image ls

1. Save all of the images into an archive file.

• All of the images can be saved as a single archive file which can be used to load all at once on our airgapped machine with the following command.

docker save -o image-tarfile.tar \
  list \
  of \
  images

Here is an an example of the command used for the images in this tutorial:

docker save -o registry/all_images.tar \
  docker.io/gitea/gitea:1.19.3 \
  quay.io/keycloak/keycloak:21.1.1 \
  ghcr.io/siderolabs/imager:v1.4.5 \
  ghcr.io/siderolabs/flannel:v0.21.4 \
  ghcr.io/siderolabs/install-cni:v1.4.0-1-g9b07505 \
  docker.io/coredns/coredns:1.10.1 \
  gcr.io/etcd-development/etcd:v3.5.9 \
  registry.k8s.io/kube-apiserver:v1.27.2 \
  registry.k8s.io/kube-controller-manager:v1.27.2 \
  registry.k8s.io/kube-scheduler:v1.27.2 \
  registry.k8s.io/kube-proxy:v1.27.2 \
  ghcr.io/siderolabs/kubelet:v1.27.2 \
  ghcr.io/siderolabs/installer:v1.4.5 \
  registry.k8s.io/pause:3.6

Move Dependencies

Now that we have all the packages necessary for the airgapped deployment of Omni, we'll create a compressed archive file and move it to our airgapped network.

The directory structure should look like this now:

airgap
├── certs
│   ├── fullchain.pem
│   ├── omni.asc
│   └── privkey.pem
├── gitea
│   └── app.ini
├── keycloak
├── omni
└── registry
    ├── omni-image.tar # Provided to you by Sidero Labs
    └── all_images.tar

Create a compressed archive file to move to our airgap machine.

cd ../
tar czvf omni-airgap.tar.gz airgap/

Now I will use scp to move this file to my machine which does not have internet access. Use whatever method you prefer to move this file.

scp omni-airgap.tar.gz $USERNAME@$AIRGAP_MACHINE:/home/$USERNAME/

Lastly, I will log in to my airgapped machine and extract the compressed archive file in the home directory

cd ~/
tar xzvf omni-airgap.tar.gz

Log in Airgapped Machine

From here on out, the rest of the tutorial will take place from the airgapped machine we will be installing Omni, Keycloak, and Gitea on.

Gitea

Install Gitea

Load the images we moved over. This will load all the images into Docker on the airgapped machine.

docker load -i registry/omni-image.tar
docker load -i registry/all_images.tar

Run Gitea using Docker:

• The app.ini file is already configured and mounted below with the - v argument.

sudo docker run -it \
    -v $PWD/certs/privkey.pem:/data/gitea/key.pem \
    -v $PWD/certs/fullchain.pem:/data/gitea/cert.pem \
    -v $PWD/gitea/app.ini:/data/gitea/conf/app.ini \
    -v $PWD/gitea/data/:/data/gitea/ \
    -p 3000:3000 \
    gitea/gitea:1.19.3

You may now log in at the https://${GITEA_HOSTNAME}:3000 to begin configuring Gitea to store all the images needed for Omni and Talos.

Gitea setup

Create a user

Click the Register button at the top right corner. The first user created will be created as an administrator - permissions can be adjusted afterwards if you like.

Create organizations

After registering an admin user, the organizations can be created which will act as the package repositories for storing images. Create the following organizations:

• siderolabs

• keycloak

• coredns

• etcd-development

• registry-k8s-io-proxy

Push Images to Gitea

Now that all of our organizations have been created, we can push the images we loaded into our Gitea for deploying Keycloak, Omni, and storing images used by Talos.

For all of the images loaded, we first need to tag them for our Gitea.

sudo docker tag original-image:tag gitea:3000/new-image:tag

For example, if I am tagging the kube-proxy image it will look like this:

NOTE: Don't forget to tag all of the images from registry.k8s.io to go to the registry-k8s-io-proxy organization created in Gitea.

docker tag registry.k8s.io/kube-proxy:v1.27.2 ${GITEA_HOSTNAME}:3000/registry-k8s-io-proxy/kube-proxy:v1.27.2

Finally, push all the images into Gitea.

docker push ${GITEA_HOSTNAME}:3000/registry-k8s-io-proxy/kube-proxy:v1.27.2

Keycloak

Install Keycloak

The image used for keycloak is already loaded into Gitea and there are no files to stage before starting it so I'll run the following command to start it. Replace KEYCLOAK_HOSTNAME and GITEA_HOSTNAME with your own hostnames.

sudo docker run -it \
    -p 8080:8080 \
    -p 8443:8443 \
    -v $PWD/certs/fullchain.pem:/etc/x509/https/tls.crt \
    -v $PWD/certs/privkey.pem:/etc/x509/https/tls.key \
    -v $PWD/keycloak/data:/opt/keycloak/data \
    -e KEYCLOAK_ADMIN=admin \
    -e KEYCLOAK_ADMIN_PASSWORD=admin \
    -e KC_HOSTNAME=${KEYCLOAK_HOSTNAME} \
    -e KC_HTTPS_CERTIFICATE_FILE=/etc/x509/https/tls.crt \
    -e KC_HTTPS_CERTIFICATE_KEY_FILE=/etc/x509/https/tls.key \
    ${GITEA_HOSTNAME}:3000/keycloak/keycloak:21.1.1 \
    start

Once Keycloak is installed, you can reach it in your browser at `https://${KEYCLOAK_HOSTNAME}:3000

Configuring Keycloak

Omni

With Keycloak and Gitea installed and configured, we're ready to start up Omni and start creating and managing clusters.

Install Omni

To install Omni, first generate a UUID to pass to Omni when we start it.

export OMNI_ACCOUNT_UUID=$(uuidgen)

Next run the following command, replacing hostnames for Omni, Gitea, or Keycloak with your own.

sudo docker run \
  --net=host \
  --cap-add=NET_ADMIN \
  -v $PWD/etcd:/_out/etcd \
  -v $PWD/certs/fullchain.pem:/fullchain.pem \
  -v $PWD/certs/privkey.pem:/privkey.pem \
  -v $PWD/certs/omni.asc:/omni.asc \
  ${GITEA_HOSTNAME}:3000/siderolabs/omni:v0.12.0 \
    --account-id=${OMNI_ACCOUNT_UUID} \
    --name=omni \
    --cert=/fullchain.pem \
    --key=/privkey.pem \
    --siderolink-api-cert=/fullchain.pem \
    --siderolink-api-key=/privkey.pem \
    --private-key-source=file:///omni.asc \
    --event-sink-port=8091 \
    --bind-addr=0.0.0.0:443 \
    --siderolink-api-bind-addr=0.0.0.0:8090 \
    --k8s-proxy-bind-addr=0.0.0.0:8100 \
    --advertised-api-url=https://${OMNI_HOSTNAME}:443/ \
    --siderolink-api-advertised-url=https://${OMNI_HOSTNAME}:8090/ \
    --siderolink-wireguard-advertised-addr=${OMNI_HOSTNAME}:50180 \
    --advertised-kubernetes-proxy-url=https://${OMNI_HOSTNAME}:8100/ \
    --auth-auth0-enabled=false \
    --auth-saml-enabled \
    --talos-installer-registry=${GITEA_HOSTNAME}:3000/siderolabs/installer \
    --talos-imager-image=${GITEA_HOSTNAME}:3000/siderolabs/imager:v1.4.5 \
    --kubernetes-registry=${GITEA_HOSTNAME}:3000/siderolabs/kubelet \
    --auth-saml-url "https://${KEYCLOAK_HOSTNAME}:8443/realms/omni/protocol/saml/descriptor"

What's going on here:

• --auth-auth0-enabled=false tells Omni not to use Auth0.

• --auth-saml-enabled enables SAML authentication.

• --talos-installer-registry, --talos-imager-image and --kubernetes-registry allow you to set the default images used by Omni to point to your local repository.

• --auth-saml-url is the URL we saved earlier in the configuration of Keycloak.

  • --auth-saml-metadata may also be used if you would like to pass it as a file instead of a URL and can be used if using self-signed certificates for Keycloak.

Creating a cluster

Guides on creating a cluster on Omni can be found here:

machine:
  registries:
    mirrors:
    docker.io:
      endpoints:
      - https://${GITEA_HOSTNAME}:3000
    gcr.io:
      endpoints:
      - https://${GITEA_HOSTNAME}:3000
    ghcr.io:
      endpoints:
      - https://${GITEA_HOSTNAME}:3000
    registry.k8s.io:
      endpoints:
      - https://${GITEA_HOSTNAME}:3000/v2/registry-k8s-io-proxy
      overridePath: true
cluster:
  discovery:
    enabled: false

Specifics on patching machines can be found here:

Closure

With Omni, Gitea, and Keycloak set up, you are ready to start managing and installing Talos clusters on your network! The suite of applications installed in this tutorial is an example of how an airgapped environment can be set up to make the most out of the Kubernetes clusters on your network. Other container registries or authentication providers may also be used with a similar setup, but this suite was chosen to give you starting point and an example of what your environment could look like.

Welcome to the Omni user guide! This guide shows you everything from getting started to more advanced deployments with Omni.

What is Omni?

Omni is a Kubernetes management platform that simplifies the creation and management of Kubernetes clusters on any environment to provide a simple, secure, and resilient platform. It automates cluster creation, management and upgrades, and integrates Kubernetes and Omni access into enterprise identity providers. While Omni does provide a powerful UI, tight integration with Talos Linux means the platform is 100% API-driven from Linux to Kubernetes to Omni.

Simple

Omni automatically creates a highly available Kubernetes API endpoint, transparently provides secure encryption, and automates Kubernetes and OS upgrades. Omni is perfectly suited for managing edge clusters or single node clusters, and in large data centers.

Omni is available as a SaaS operated by Sidero Labs, and for licensing for on-premises installations.

Enterprise support and fully managed and operated plans are available.

Secure

Omni creates clusters with both Kubernetes and the OS configured for best-practices security. All traffic to Omni is wireguard-encrypted. Optionally, traffic between the cluster nodes can be encrypted, allowing clusters to span insecure networks. Integration with enterprise identity providers ensures that even admin-level kubeconfig is validated against current user access-lists.

Is Omni for me?

Omni is excellent for managing clusters in just about any environment you have. Machines in the cloud, on-premise, edge, home - they all can be managed with Omni. With Omni you can even create hybrid clusters consisting of machines in disparate locations around the world.

Some common use cases are:

• On-premise bare metal clusters that can be scaled up with machines in the cloud

• Edge clusters that are supported by machines in the data center and/or cloud

• Mixed cloud

• Single node edge clusters

Ready to get started?

Status

You can subscribe to updates by clicking the Bell icon in the top right corner.

Security notifications

If you would like to watch a quick video of the process you can follow along here.

Prerequisites

Network access

If your machines have outgoing internet access, you are all set. Machines should have access to the Wireguard Endpoint shown on the Omni Home panel, which lists the IP address and port (e.g. 199.99.99.100:10001) that machines will connect to. Machines need to be able to reach that address both on the UDP port specified, and on TCP port 443.

Create virtual or boot physical machines

The simplest way to experience Omni is to create some virtual machines. We suggest any virtualization platform that can boot off an ISO (UTM, ProxMox, VMware Fusion, etc) although any cloud platform can also be used with minor adjustments. Bare metal can boot from a physical CD, USB drive, virtual media, or PXE.

talosctl

talosctl is the command line tool for managing Talos Linux via the management API, but when machines connect to Omni it is not required. Instead cluster management is done via the Omni UI or omnictl. We still recommend installing talosctl to investigate the state of the nodes and explore functionality.

Download talosctl, kubectl, kubectl-oidc_login, and omnictl (macOS and Linux):

brew install siderolabs/tap/sidero-tools

Make sure you download the talosconfig and omniconfig files from the Omni web interface and place them in the default locations ~/.talos/config and ~/.config/omni/config/ respectively.

Download Installation Media

Omni is a BYO Machine platform - all you need to do is boot your machines (virtual or physical) off an Omni image. To download the installation media, go to the Home Screen in Omni, and select "Download Installation Media" from the right hand side. Select the appropriate media and platform type.

Images exist for many platforms, but you will have to follow the specific installation instructions for that platform (which often involve copying the image to S3 type storage, creating a machine image from it, etc.)

Boot machines off the downloaded image

Create at least 1 virtual machine with 2GB of memory (4GB or more is recommended) using your Hypervisor. Have each virtual machine boot off the ISO image you just downloaded.

After a few seconds, the machines should show in the Machines panel of Omni, with the available tag. They will also have tags showing their architecture, memory, cores and other information.

Create Cluster

Click "Clusters" on the left navigation panel, then "Create Cluster" in the top right. You can give your cluster a name, select the version of Talos Linux to install, and the version of Kubernetes. You can also specify any patches that should be applied to your cluster, but in most cases these are not needed to get started.

In the section headed "Available Machines", select at least one machine to be the control plane, by clicking CP. You will want an odd number of control plane nodes (e.g. 1, 3, 5). Select one machine to be a worker, by clicking W0 next to the machine.

Then click Create Cluster. Your cluster is now being created, and you will be taken to the Cluster Overview page. From this page you can download the kubeconfig and talosconfig files for your cluster, by clicking the buttons on the right hand side.

Please note that because Omni manages the state of the Talos nodes, and protects the security of the Kubernetes and Talos credentials. Because of this, some talosctl commands (such as talosctl reset) will return PermissionDenied on Omni managed clusters - such operations must be done through the Omni UI or API calls.

Access Kubernetes

You can query your Kubernetes cluster using normal Kubernetes operations:

kubectl --kubeconfig ./talos-default-kubeconfig.yaml get nodes

Note: you will have to change the referenced kubeconfig file depending on the name of the cluster you created.

The first time you use the kubectl command to query a cluster, a browser window will open requiring you to authenticate with your identity provider (Google or GitHub most commonly.) If you get a message error: unknown command "oidc-login" for "kubectl" Unable to connect to the server: then you need to install the oidc-login plugin as noted above.

Access Talos commands

You can explore Talos API commands. The first time you access the Talos API, a browser window will start to authenticate your request. The downloaded talosconfig file for the cluster includes the Omni endpoint, so you do not need to specify endpoints, just nodes.

talosctl --talosconfig ./talos-default-talosconfig.yaml --nodes 10.5.0.2 get members

You will need to change the name of the talosconfig file, if you changed the cluster name from the default; and also use the actual IP or name of the nodes you created (which are shown in Omni) in place of the node IP.

Note that because Omni manages the state of the Talos nodes, and protects the security of the Kubernetes and Talos credentials, some talosctl commands (such as talosctl reset) will return PermissionDenied on Omni managed clusters - such operations must be done through the Omni UI or API calls.

Explore Omni

Now you have a complete cluster, with a high-availability Kubernetes API endpoint running on the Omni infrastructure, where all authentication is tied in to your enterprise identity provider. It's a good time to explore all that Omni can offer, including other areas of the UI such as:

• etcd backup and restores

• simple cluster upgrades of Kubernetes and Operating System

• proxying of workload HTTP access

• simple scaling up and down of clusters

• the concept of Machine Sets, that let you manage your infrastructure by classes

Destroy the Cluster

When you are all done, you can remove the cluster by clicking "Destroy Cluster", in the bottom right of the Cluster Overview panel. This will wipe the machines and return them to the Available state.

Cluster example

Components

The contrib example includes:

Use

sed -i 's|https://github.com/siderolabs/contrib.git|<your-git-repo>|' apps/argocd/argocd/bootstrap-app-set.yaml
kustomize build apps/argocd/argocd | yq -i 'with(.cluster.inlineManifests.[] | select(.name=="argocd"); .contents=load_str("/dev/stdin"))' infra/patches/argocd.yaml

With these changes made you should commit the new values and push them to the git repo.

Once machines are registered you can create the cluster using the cluster template in the infra directory.

omnictl cluster template sync --file cluster-template.yaml

In this tutorial we will use SAML and ACLs to control fine-grained access to Kubernetes clusters.

Let's assume that at our organization:

• We run a Keycloak instance as the SAML identity provider.

• Have our Omni instance already configured to use Keycloak as the SAML identity provider.

• Our Omni instance has 2 types of clusters:

  • Staging clusters with the name prefix staging-: staging-1, staging-2, etc.

  • Production clusters with the name prefix prod-: prod-1, prod-2, etc.

• We want the users with the SAML role omni-cluster-admin to have full access to all clusters.

• We want the users with the SAML role omni-cluster-support to have full access to staging clusters and read-only access to production clusters.

Sign in as the initial SAML User

If our Omni instance has no users yet, the initial user who signs in via SAML will be automatically assigned to the Omni Admin role.

We sign in as the user admin@example.org and get the Omni Admin role.

Configuring the AccessPolicy

We need to configure the ACL to assign the omni-cluster-support role to the users with the SAML role omni-cluster-support and the omni-cluster-admin role to the users with the SAML role omni-cluster-admin.

Create the following YAML file acl.yaml:

metadata:
  namespace: default
  type: AccessPolicies.omni.sidero.dev
  id: access-policy
spec:
  usergroups:
    support:
      users:
        - labelselectors:
            - saml.omni.sidero.dev/role/omni-cluster-support=
    admin:
      users:
        - labelselectors:
            - saml.omni.sidero.dev/role/omni-cluster-admin=
  clustergroups:
    staging:
      clusters:
        - match: staging-*
    production:
      clusters:
        - match: prod-*
    all:
      clusters:
        - match: "*"
  rules:
    - users:
        - group/support
      clusters:
        - group/staging
      role: Operator
    - users:
        - group/support
      clusters:
        - group/production
      role: Reader
      kubernetes:
        impersonate:
          groups:
            - read-only
    - users:
        - group/admin
      clusters:
        - group/all
      role: Operator
  tests:
    - name: support engineer has Operator access to staging cluster
      user:
        name: support-eng@example.org
        labels:
          saml.omni.sidero.dev/role/omni-cluster-support: ""
      cluster:
        name: staging-1
      expected:
        role: Operator
    - name: support engineer has Reader access to prod cluster and impersonates read-only group
      user:
        name: support-eng@example.org
        labels:
          saml.omni.sidero.dev/role/omni-cluster-support: ""
      cluster:
        name: prod-1
      expected:
        role: Reader
        kubernetes:
          impersonate:
            groups:
              - read-only
    - name: admin has Operator access to staging cluster
      user:
        name: admin-1@example.org
        labels:
          saml.omni.sidero.dev/role/omni-cluster-admin: ""
      cluster:
        name: staging-1
      expected:
        role: Operator
    - name: admin has Operator access to prod cluster
      user:
        name: admin-1@example.org
        labels:
          saml.omni.sidero.dev/role/omni-cluster-admin: ""
      cluster:
        name: prod-1
      expected:
        role: Operator

As the admin user admin@example.org, apply this ACL using omnictl:

$ omnictl apply -f acl.yaml

Accessing the Clusters

Now, in an incognito window, log in as a support engineer, cluster-support-1@example.org. Since the user is not assigned to any Omni role yet, they cannot use Omni Web.

Download omnictl and omniconfig from the UI, and try to list the clusters by using it:

$ omnictl --omniconfig ./support-omniconfig.yaml get cluster
NAMESPACE   TYPE   ID   VERSION
Error: rpc error: code = PermissionDenied desc = failed to validate: 1 error occurred:
	* rpc error: code = PermissionDenied desc = unauthorized: access denied: insufficient role: "None"

You won't be able to list the clusters because the user is not assigned to any Omni role.

Now try to get the cluster staging-1:

$ omnictl --omniconfig ./support-omniconfig.yaml get cluster staging-1
NAMESPACE   TYPE      ID          VERSION
default     Cluster   staging-1   5

You can get the cluster staging-1 because the ACL allows the user to access the cluster.

Finally, try to delete the cluster staging-1:

$ omnictl --omniconfig ./support-omniconfig.yaml delete cluster staging-1
torn down Clusters.omni.sidero.dev staging-1
destroyed Clusters.omni.sidero.dev staging-1

The operation will succeed, because the ACL allows Operator-level access to the cluster for the user.

Try to do the same operations with the cluster prod-1:

$ omnictl --omniconfig ./support-omniconfig.yaml get cluster prod-1
NAMESPACE   TYPE      ID          VERSION
default     Cluster   prod-1   5

$ omnictl --omniconfig ./support-omniconfig.yaml delete cluster prod-1
Error: rpc error: code = PermissionDenied desc = failed to validate: 1 error occurred:
	* rpc error: code = PermissionDenied desc = unauthorized: access denied: insufficient role: "Reader"

The user will be able to get the cluster but not delete it, because the ACL allows only Reader-level access to the cluster for the user.

If you do the same operations as the admin user, you'll notice that you are able to both get and delete staging and production clusters.

Assigning Omni roles to Users

If you want to allow SAML users to use Omni Web, you need to assign them at least the Reader role. As the admin, sign in to Omni Web and assign the role Reader to both cluster-support-1@example.org and cluster-admin-1@example.org.

Now, as the support engineer, you can sign out & sign in again to Omni Web and see the clusters staging-1 and prod-1 in the UI.

1. Log in to Otka

2. Create a new App Integration

3. Select “SAML 2.0”

4. Give the Application a recognizable name (we suggest simply “Omni”)

5. Set the SAML Settings and Attribute Statements as shown below:

1. Click “Next” and optionally fill out the Feedback, then click “Finish”

Once that is complete, you should now be able to open the “Assignments” tab for the application you just created and manage your users and access as usual.

Requirements

• An Omni instance (either on the Sidero Labs SaaS or self-hosted) with Admin access

• Some bare-metal machines with:

  • BMC (baseboard management controller) power management capabilities via one of the following:

    • IPMI

    • Redfish

  • Outbound access to the Omni instance and the Image Factory

• A machine/container/cluster etc. in the same subnet as the bare-metal machines, in order to run the infrastructure provider service

In this tutorial, we will assume:

• Our managed Omni instance is running at my-instance.omni.siderolabs.io

• We will use the public Image Factory at factory.talos.dev

• Bare-Metal machines with IPMI support

• An additional server with Docker installed to run the infrastructure provider service, with the IP address 172.16.0.42 within the subnet 172.16.0.0/24, reachable by the bare-metal machines

• Two bare-metal servers within the subnet 172.16.0.0/24 with access to the infrastructure provider, our Omni instance, and to the Image Factory

1. Creating an Omni service account

We start by creating an Omni service account for the infrastructure provider to authenticate/authorize to Omni.

2. Starting the provider

We will run the provider in a Docker container in our server with IP 172.16.0.42.

At the time of writing, it is ghcr.io/siderolabs/omni-infra-provider-bare-metal:v0.1.0, and this is what we shall use in this tutorial.

Set the required environment variables, using the service account key from the previous step:

Run the following command to start the provider service:

Make sure it is running by checking its status:

Sample output:

Start tailing its logs in a separate shell:

Sample output:

At this point, the provider has started and established a connection to our Omni instance.

3. Starting the Bare-Metal Machines

At this point, we can boot our bare-metal machines. Before we start, make sure that they are configured to boot over the network via PXE on the next boot, so that they can be booted by the provider.

Power cycle the machines, and when they attempt to boot via PXE, you can see in the provider logs that they have been PXE booted by the provider. Log messages should be similar to the lines below:

4. Configuring and Accepting the Machines in Omni

At this point, the machines should be booted into the Agent Mode, and have established a SideroLink connection to our Omni instance.

4.1. Verifying the Machines

Let's verify our machines:

Our machines have the following IDs:

• 33313750-3538-5a43-4a44-315430304c46

• 33313750-3538-5a43-4a44-315430304c47

For security reasons, the machines cannot be provisioned in Omni before they are "Accepted". We will accept these machines using the Omni API.

4.2. Optional: Providing BMC (e.g., IPMI/Redfish) Configuration Manually

Normally, when we accept a machine in Omni, the provider will auto-configure the BMC configuration, such as the IPMI IP, username and password automatically by asking the agent service running on the Talos machine.

Sometimes we do not want this behavior - instead, we already have this information at hand, and want to provide it manually, skipping this step.

To do this, before accepting these machines, we need to create a resource with these credentials in Omni.

Create a bmc-config.yaml file with the following contents:

Fill it in with the information of your machine ID and IPMI configuration.

As an Omni admin, create this resource using omnictl:

4.3. Accepting the Machines

As the final step, we need to accept these machines in Omni.

When you do this, the provider will do the following under the hood:

• Ask the Talos Agent service on the machines to configure their IPMI credentials

• Retrieve these credentials and store them

• Wipes the disks of these machines

• Power off these machines over IPMI

Additionally, Omni will create a Machine, and an InfraMachineStatus resource for each machine. You can verify this by:

Output will be similar to:

5. Adding Machines to a Cluster

When you add these machines to a cluster, the following will happen under the hood.

The provider will:

• Power these machines on, marking their next boot to be a PXE boot

• PXE boot them into Talos maintenance mode

Then Omni will proceed with the regular flow of:

• Applying a configuration to the machine, causing Talos to be installed to the disk

• Reboot (possibly using kexec)

The cluster will be provisioned as normal, and will get to the Ready status.

6. Removing Machines from a Cluster

When you delete a cluster and/or remove some bare-metal machines from a cluster, the following will happen:

Omni does the regular de-allocation flow:

• Remove the nodes from the cluster (leave etcd membership for control planes)

• Reset the machines

Afterwards, the provider will follow with these additional steps:

• PXE boot the machine into Agent Mode (to be able to wipe its disks)

• Wipe its disks

• Power off the machine

At this point, these machines will again be ready to be allocated to a different cluster.

This guide shows you how to configure your Omni instance so that new users logging in with SAML authentication are automatically assigned to a role based on their SAML role attributes.

Create the file assign-operator-to-engineers-label.yaml for the SAMLLabelRule resource, with the following content:

Copy

As an admin, create it on your Omni instance using omnictl:

Copy

This will create a resource that assigns the Operator role to any user that logs in with SAML and has the SAML attribute Role with the value engineers.

Log in to Omni as a new SAML user with the SAML attribute with name Role and value engineers.

This will cause the user created on the Omni side to be labeled as saml.omni.sidero.dev/role/engineers.

This label will match the SAMLLabelRule resource we created above, and the user will automatically be assigned the Operator role.

Note

When there are multiple matches from different SAMLLabelRule resources, the matched role with the highest access level will be assigned to the user.

Warning

This role assignment will only work for the new users logging in with SAML.

The SAML users who have already logged in to Omni at least once will not be matched by the SAMLLabelRule resource and their roles will not be updated.

Warning

If the logged in SAML user is the very first user logging in to an Omni instance, it will not be matched by the SAMLLabelRule resource and always be assigned the Admin role.

This guide shows you how to create a user in an Omni instance with SAML authentication enabled.

Grant the new user access to Omni in your SAML identity provider. The new user should login to the new user account, in order for Omni to have the account synchronized with the SAML provider.

Log into Omni using another account with Admin permissions, navigate to Settings, then find the newly added user in the list of users. Now, select “Edit User” from the menu under the ellipsis:

Change the role to Reader, Operator or Admin as appropriate:

And click “Update User”. The user will now have the appropriate role within Omni, and be associated with the SAML user.

In the Azure portal, click "Enterprise Applications".

Click "New Application" and search for "Entra SAML Toolkit".

Name this application something more meaningful if desired and click "Create".

Under the "Manage" section of the application, select "Single sign-on", then "SAML" as the single sign-on method.

In section 1 of this form, enter identifier, reply, and sign on URLs that match the following and save:

• Identifier (Entity ID): https://<domain name for omni>/saml/metadata

• Reply URL (Assertion Consumer Service URL): https://<domain name for omni>/saml/acs

• Sign on URL: https://<domain name for omni>/login

From section 3, copy the "App Federation Metadata Url" for later use.

Again, under the "Manage" section of the application, select "Users and groups".

Add any users or groups you wish to give access to your Omni environment here.

The first step to creating and managing Kubernetes clusters in Omni is registering the machines you wish to use. The machines can run anywhere Talos Linux runs. The guides in this section walk you through the specifics of different platforms.

In general, the process to register a machine with Omni consists of downloading the installation media from your Omni account, and booting the machine off that media.

To start the process, from the Home screen, click "Download Installation Media" on the right hand side:

The Download Installation Media form will open:

This allows you to select media specific to various platforms (AWS, Azure, bare metal, etc), and also to specify options, such as:

• the versions of Talos Linux to initially boot with (note that it is recommended to have all machines that will form a cluster be at the same Talos Linux version.)

• optional extensions to be bundled into the image (providing support for additional features, such as GPUs, driver microcode, etc.)

• specific machine labels to be applied to the machines booted off this image (useful for identifying machines by racks, or location, etc)

• optional capabilities such as:
• Generating iPXE boot URLs: if your machines can boot via iPXE, you can configure them to boot directly off Omni via the Image Factory integration. Simply click to generate the PXE Boot URL, and pass that in to your systems to boot from.

Note that some platforms have other specific requirements, which can be seen in the articles within this section.

This guide shows you how to register a bare metal machine with Omni by PXE/iPXE booting.

Copy the Required Kernel Parameters

Upon logging in you will be presented with the Omni dashboard. Click the “Copy Kernel Parameters” button on the right hand side, and save the value for later.

Download the PXE/iPXE Assets

The following example assumes the use of Matchbox server.

Create the Profile

Place the following in /var/lib/matchbox/profiles/default.json:

{
  "id": "default",
  "name": "default",
  "boot": {
    "kernel": "/assets/vmlinuz",
    "initrd": ["/assets/initramfs.xz"],
    "args": [
      "initrd=initramfs.xz",
      "init_on_alloc=1",
      "slab_nomerge",
      "pti=on",
      "console=tty0",
      "console=ttyS0",
      "printk.devkmsg=on",
      "talos.platform=metal",
      "siderolink.api=<your siderolink.api value>",
      "talos.events.sink=<your talos.events.sink value>",
      "talos.logging.kernel=<your talos.logging.kernel value>"
    ]
  }
}

Update siderolink.api, talos.events.sink, and talos.logging.kernel with the kernel parameters copied from the dashboard.

Place the following in /var/lib/matchbox/groups/default.json:

Create the Group

{
  "id": "default",
  "name": "default",
  "profile": "default"
}

Once your machine is configured to PXE boot using your tool of choice, power the machine on.

Conclusion

Navigate to the “Machines” menu in the sidebar of Omni. You should now see a machine listed.

You now have a bare metal machine registered with Omni and ready to provision.

Unifi Identity Enterprise

This section describes how to use Unifi Identity Enterprise (here forward UIIE) SSO with Omni via SAML

First, login to the UIIE Manager portal and navigate to the SSO Apps section in the left menu.

Next, Add a new app. Choose “Add Custom App”

Next, click Add on the “SAML 2.0” option for Sign-on Method

You’ll now be in the Add SAML 2.0 App screen where we’ll define the app.

After entering the above values and clicking the “Add” button, you’ll be taken to another screen with some details. We don’t need anything from here, we’ll get info we need later after further configuration, so just click “Done” to proceed.

You’ll now be on the screen to manage the app, here you’ll want to assign users/groups according to who you would like to have the ability to login to Omni. To start with, you probably only want to assign the person who will be the primary admin, as the first user to login will be granted that role in Omni. Therefore, best practice would be to assign your primary admin, have them login to Omni, then come back into the app here and assign any other users who should have access.

Once you’ve assigned the user(s) accordingly, click the “Settings” bubble at the top of the screen as some final configuration is needed here.

Expand the “Sign On” section at the bottom of the settings page via the “Show More” down arrow.

At the bottom of this section, you’ll see an “Attibute Statements” block, here the mappings from UIIE to Omni fields needs to be entered as below. Use the “Add Another” button to create new ones.

Lastly, you’ll need the IDP Metadata file which can be obtained via the View Setup Instructions link or downloaded as an xml file via the Identity Provider metadata link; both of which are slightly further up the page.

A copy of this file needs to be on the host which will run the Omni container as we’ll feed it in to the container at runtime. You can copy paste contents or download/upload the file whichever is easiest. For the remainder of this guide, we’ll assume this file ends up at the following location on your container host: ~/uiieIDPmetadata.xml

This completes the configurations required in UIIE

Omni

To get Omni to use UIIE as a SAML provider, the following flags should be passed to Docker & the Omni container on launch.

For example;

Copy

Unfortunately UIIE does not expose group attributes, so you will have to manually assign Omni groups/roles to the users as they are created on first login.

Set your AWS region

REGION="us-west-2"

Creating the subnet

First, we need to know what VPC to create the subnet on, so let’s describe the VPCs in the region where we want to create the Omni machines.

aws ec2 describe-vpcs --region $REGION

Output:

{
    "Vpcs": [
        {
            "CidrBlock": "172.31.0.0/16",
            "DhcpOptionsId": "dopt-0238fea7541672af0",
            "State": "available",
            "VpcId": "vpc-04ea926270c55d724",
            "OwnerId": "753518523373",
            "InstanceTenancy": "default",
            "CidrBlockAssociationSet": [
                {
                    "AssociationId": "vpc-cidr-assoc-0e518f7ac9d02907d",
                    "CidrBlock": "172.31.0.0/16",
                    "CidrBlockState": {
                        "State": "associated"
                    }
                }
            ],
            "IsDefault": true
        }
    ]
}

Note the VpcId (vpc-04ea926270c55d724).

Now, create a subnet on that VPC with a CIDR block that is within the CIDR block of the VPC. In the above example, as the VPC has a CIDR block of 172.31.0.0/16, we can use 172.31.128.0/20.

$ aws ec2 create-subnet \
    --vpc-id vpc-04ea926270c55d724 \
    --region us-west-2 \
    --cidr-block 172.31.128.0/20

Output:

{
    "Subnet": {
        "AvailabilityZone": "us-west-2c",
        "AvailabilityZoneId": "usw2-az3",
        "AvailableIpAddressCount": 4091,
        "CidrBlock": "172.31.192.0/20",
        "DefaultForAz": false,
        "MapPublicIpOnLaunch": false,
        "State": "available",
        "SubnetId": "subnet-04f4d6708a2c2fb0d",
        "VpcId": "vpc-04ea926270c55d724",
        "OwnerId": "753518523373",
        "AssignIpv6AddressOnCreation": false,
        "Ipv6CidrBlockAssociationSet": [],
        "SubnetArn": "arn:aws:ec2:us-west-2:753518523373:subnet/subnet-04f4d6708a2c2fb0d",
        "EnableDns64": false,
        "Ipv6Native": false,
        "PrivateDnsNameOptionsOnLaunch": {
            "HostnameType": "ip-name",
            "EnableResourceNameDnsARecord": false,
            "EnableResourceNameDnsAAAARecord": false
        }
    }
}

Note the SubnetID (subnet-04f4d6708a2c2fb0d).

Create the Security Group

aws ec2 create-security-group \
    --region $REGION \
    --group-name omni-aws-sg \
    --description "Security Group for Omni EC2 instances"

Output

{   
     "GroupId": "sg-0b2073b72a3ca4b03"
}

Note the GroupId (sg-0b2073b72a3ca4b03).

Allow all internal traffic within the same security group, so that Kubernetes applications can talk to each other on different machines:

aws ec2 authorize-security-group-ingress \
    --region $REGION \
    --group-name omni-aws-sg \
    --protocol all \
    --port 0 \
    --source-group omni-aws-sg

Creating the bootable AMI

To do so, log in to your Omni account, and, from the Omni overview page, select “Download Installation Media”. Select “AWS AMI (amd64)” or “AWS AMI (arm64)”, as appropriate for your desired EC2 instances. (Most are amd64.) Click “Download”, and the AMI will be downloaded to you local machine.

Extract the downloaded aws-amd64.tar.gz Then copy the disk.raw file to S3. We need to create a bucket, copy the image file to it, import it as a snapshot, then register an AMI image from it.

Create S3 bucket

REGION="us-west-2"
aws s3api create-bucket \
    --bucket <bucket name> \
    --create-bucket-configuration LocationConstraint=$REGION \
    --acl private

Copy image file to the bucket

aws s3 cp disk.raw s3://<bucket name>/omni-aws.raw

Import the image as a snapshot

aws ec2 import-snapshot \
    --region $REGION \
    --description "Omni AWS" \
    --disk-container "Format=raw,UserBucket={S3Bucket=<bucket name>,S3Key=omni-aws.raw}"

Output:

{
    "Description": "Omni AWS",
    "ImportTaskId": "import-snap-1234567890abcdef0",
    "SnapshotTaskDetail": {
        "Description": "Omni AWS",
        "DiskImageSize": "0.0",
        "Format": "RAW",
        "Progress": "3",
        "Status": "active",
        "StatusMessage": "pending"
        "UserBucket": {
            "S3Bucket": "<bucket name>",
            "S3Key": "omni-aws.raw"
        }
    }
}

Check the status of the import with:

aws ec2 describe-import-snapshot-tasks \
    --region $REGION \
    --import-task-ids

Output:

While the import is in progress:
{
    "ImportSnapshotTasks": [
        {
            "Description": "Omni AWS",
            "ImportTaskId": "import-snap-1234567890abcdef0",
            "SnapshotTaskDetail": {
                "Description": "Omni AWS",
                "DiskImageSize": "705638400.0",
                "Format": "RAW",
                "Progress": "42",
                "Status": "active",
                "StatusMessage": "downloading/converting",
                "UserBucket": {
                    "S3Bucket": "<bucket name>",
                    "S3Key": "omni-aws.raw"
                }
            }
        }
    ]
}

Once completed:
{
    "ImportSnapshotTasks": [
        {
            "Description": "Omni AWS",
            "ImportTaskId": "import-snap-1234567890abcdef0",
            "SnapshotTaskDetail": {
                "DiskImageSize": 8589934592.0,
                "Format": "raw",
                "SnapshotId": "snap-056a7e086ecf7b452",
                "Status": "completed",
                "UserBucket": {
                    "S3Bucket": "<bucket name>",
                    "S3Key": "omni-aws.raw"
                }
            },
            "Tags": []
        }
    ]
}

Once the Status is completed note the SnapshotId (snap-056a7e086ecf7b452).

Register the Image

aws ec2 register-image \
    --region $REGION \
    --block-device-mappings "DeviceName=/dev/xvda,VirtualName=talos,Ebs={DeleteOnTermination=true,SnapshotId=$SNAPSHOT,VolumeSize=8,VolumeType=gp2}" \
    --root-device-name /dev/xvda \
    --virtualization-type hvm \
    --architecture x86_64 \
    --ena-support \
    --name omni-aws-ami

Output:

{
    "ImageId": "ami-07961b424e87e827f"
}

Note the ImageId (ami-07961b424e87e827f).

Create EC2 instances from the AMI

Now, using the AMI we created, along with the security group created above, provision EC2 instances:

 aws ec2 run-instances \
    --region  $REGION \
    --image-id ami-07961b424e87e827f \
    --count 1 \
    --instance-type t3.small   \
    --subnet-id subnet-0a7f5f87f62c301ea \
    --security-group-ids $SECURITY_GROUP   \
    --associate-public-ip-address \
    --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=omni-aws-ami}]" \
    --instance-market-options '{"MarketType":"spot"}'

Dashboard

Upon logging in you will be presented with the Omni dashboard.

Download the Hetzner Image

First, download the Hetzner image from the Omni portal by clicking on the “Download Installation Media” button. Now, click on the “Options” dropdown menu and search for the “Hetzner” option. Notice there are two options: one for amd64 and another for arm64. Select the appropriate option for the machine you are registering. Now, click the “Download” button.

packer {
  required_plugins {
    hcloud = {
      version = ">= 1.0.0"
      source  = "github.com/hashicorp/hcloud"
    }
  }
}

locals {
  image = "<path to downloaded installation media>"
}

source "hcloud" "talos" {
  rescue       = "linux64"
  image        = "debian-11"
  location     = "hel1"
  server_type  = "cx11"
  ssh_username = "root"

  snapshot_name = "Omni Image"
}

build {
  sources = ["source.hcloud.talos"]

  provisioner "file" {
    source = "${local.image}"
    destination = "/tmp/talos.raw.xz"
  }

  provisioner "shell" {
    inline = [
      "xz -d -c /tmp/talos.raw.xz | dd of=/dev/sda && sync",
    ]
  }
}

export HCLOUD_TOKEN=${TOKEN}
packer init .
packer build .

Create a Server

hcloud context create talos

hcloud server create --name omni-talos-1 \
    --image <image ID> \
    --type cx21 --location <location>

Warning

Conclusion

Navigate to the “Machines” menu in the sidebar. You should now see a machine listed.

You now have a Hetzner server registered with Omni and ready to provision.

This guide shows you how to create a cluster from registered machines.

1. Installation

Pick one of the installation methods below.

a. Recommended installation

To install omnictland the kubectl oidc-loginplugin automatically on macOS and Linux you can use the provided Hombrew formula.

brew install siderolabs/tap/sidero-tools

This will install talosctl, omnictl, and kube-oidc_loginand keep them updated.

b. Manual installation

Download omnictl from the Omni dashboard.

mv omnictl-darwin-arm64 omnictl
chmod +x omnictl
mv omnictl /usr/local/bin

2. Configuration File

Download omniconfig from the Omni dashboard.

Omni configuration is stored under $XDG_CONFIG_HOME/omni/config. This means, the correct configuration file locations for omniconfig are the following:

Add the downloaded omniconfig.yaml to the default location for your OS to use it with omnictl:

cp omniconfig.yaml <LOCATION_FOR_YOUR_OS_ABOVE>

If you would like to merge the omniconfig.yaml with an existing configuration, use the following command:

omnictl config merge ./omniconfig.yaml

List the contexts to verify that the omniconfig was added:

$ omnictl config contexts
CURRENT   NAME         URL
          ...
          example      https://example.omni.siderolabs.io/
          ...

Run omnictl for the first time to perform initial authentication using a web browser:

omnictl get clusters

If the browser window does not open automatically, it can be opened manually by copying and pasting the URL into a web browser:

BROWSER=echo omnictl get clusters

Recommended

brew install siderolabs/tap/sidero-tools

This will also keep your version of talosctl up to date with new releases. This homebrew tap also has formulae for omnictl if you need to install that package.

Manual installation

curl -sL https://talos.dev/install | sh

You now have talosctl installed.

Please note that because Omni manages the state of the Talos nodes, and protects the security of the Kubernetes and Talos credentials, some talosctl commands (such as talosctl reset) will return PermissionDenied on Omni managed clusters - such operations must be done through the Omni UI or API calls.

Windows installation

Windows binaries can be downloaded from GitHub releases for Talos and Omni.

Omni allows you to create patches and target the patches to all members of a cluster; all control plane nodes; all worker nodes; or specific machines. Upon logging in, click the “Clusters” menu item on the left. Now, select “Config Patches” from the menu under the ellipsis:

(You can also navigate to the specific cluster, and then select Config Patches in the right hand menu.)

Next, click “Create Patch”:

Pick the specific machine or class of machines from the “Patch Target” dropdown:

Type in the desired config patch:

Click “Save” to create the config patch.

Omni is available as a hosted version operated by Sidero Labs, but can also be self-hosted. This is appropriate for:

• air-gapped deployments

• industries with strict security and compliance concerns

• home use.

If you are running a self-hosted version of Omni licensed under the BSL, please regularly update to the latest release - we suggest at least monthly. Bug fixes will not be backported to older versions of Omni, so support that involves a bug fix will require an update.

Talos Linux Versions Supported

Each version of Omni will support versions of Talos Linux where the first stable release of the minor version of Talos Linux was within 18 months of the Omni release date. For example, because Talos Linux 1.3.0 was released on 2022-12-15, the Omni version released on Jun 30th, 2024 will not support any version of 1.3.x, even though patch releases of 1.3 were made within the prior 18 months. However, all versions of Talos Linux Talos 1.4 and later (which was released on 2023-04-18) are supported.

Current minimum Talos Linux support within Omni

Talos Linux 1.4

Effect of non-supported Talos Linux versions

If you neglect to update Talos Linux to a version where the initial minor version was released within the prior 18 months, your cluster will be out of support, and you must upgrade in order to receive support.

Clusters running unsupported versions of Talos Linux or Kubernetes may have limited ability to manage their clusters within Omni - clusters may not be able to have new nodes provisioned to scale up the cluster, for example.

We encourage all users to take advantage of how easy Omni makes Talos Linux and Kubernetes upgrades, to ensure they are always running the latest security and bug fixes.

This guide will show how to give the user support@example.com full access to the staging cluster but limited access to the production cluster.

Create an AccessPolicy resource

Create a local file acl.yaml:

metadata:
  namespace: default
  type: AccessPolicies.omni.sidero.dev
  id: access-policy
spec:
  rules:
    - users:
        - support@example.com
      clusters:
        - staging
      role: Operator
      kubernetes:
        impersonate:
          groups:
            - system:masters
    - users:
        - support@example.com
      clusters:
        - production
      role: Reader
      kubernetes:
        impersonate:
          groups:
            - my-app-read-only
  tests:
    - name: support engineer has full access to staging cluster
      user:
        name: support@example.com
      cluster:
        name: staging
      expected:
        role: Operator
        kubernetes:
          impersonate:
            groups:
              - system:masters
    - name: support engineer has read-only access to my-app namespace in production cluster
      user:
        name: support@example.com
      cluster:
        name: production
      expected:
        role: Reader
        kubernetes:
          impersonate:
            groups:
              - my-app-read-only

As an Omni admin, apply this ACL using omnictl:

omnictl apply -f acl.yaml

When users interact with Omni API or UI, they will be assigned to the role specified in the ACL.

When users access the Kubernetes cluster through Omni, they will have the groups specified in the ACL.

Kubernetes RBAC then can be used to grant permissions to these groups.

Create Kubernetes RBAC resources

Locally, create rbac.yaml with a Namespace called my-app, and a Role & RoleBinding to give access to the my-app-read-only group:

apiVersion: v1
kind: Namespace
metadata:
  name: my-app
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: read-only
  namespace: my-app
rules:
  - apiGroups: ["", "extensions", "apps", "batch", "autoscaling"]
    resources: ["*"]
    verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-only
  namespace: my-app
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: read-only
subjects:
  - kind: Group
    name: my-app-read-only
    apiGroup: rbac.authorization.k8s.io

As the cluster admin, apply the manifests to the Kubernetes cluster production:

kubectl apply -f rbac.yaml

Test the access

Try to access the cluster with a kubeconfig generated by the user support@example.com:

kubectl get pods -n my-app

The user should be able to list pods in the my-app namespace because of the Role and RoleBinding created above.

Try to list pods in another namespace:

kubectl get pod -n default

The user should not be able to list pods in namespace default.

Workspace ONE Access

This section describes how to create a Web App inside Workspace ONE Acces (WSOA).

First, login to the WSOA user interface and browse to Resources -> Web Apps -> New

Next, enter values for the following options before clicking on Next.

On the Single Sign-On page, enter the following values:

Still on the Single Sign-On page, in the Advanced Properties section, set the following toggle buttons;

At the bottom of the Single Sign-On page, in the Custom Attribute Mapping section, add the following attributes:

Click Next to continue and select the access policy as required by your organization.

Now it’s time to click the Save & Assign button and permit the Users and Groups allowed to login to Omni.

On the Assign screen, enter the following:

• Select the permitted group from your backing Active Directory or LDAP server.

• Set the Deployment Type to Automatic.

Finally, obtain the IdP Metadata URL by clicking on Settings and then the Copy URL link.

Omni

Provide the following flags to the Omni container on launch.

For example;

Copy

--auth-saml-enabled=true
--auth-saml-url=https://{workspace-one-host}/SAAS/API/1.0/GET/metadata/idp.xml
--auth-saml-label-rules='{"groups": "groups"}'

Note that when using groups, the group name is prefixed with saml.omni.sidero.dev/groups/ instead of role. For example;

Copy

metadata:
  namespace: default
  type: SAMLLabelRules.omni.sidero.dev
  id: assign-admin-to-platform-admins-label
spec:
  assignroleonregistration: Operator
  matchlabels:
    - saml.omni.sidero.dev/groups/omni-platform-administrators

With an Omni managed cluster, you use kubectl as with any other Kubernetes cluster, with the caveat that you must use the kubeconfig file that you download from Omni, and have the OIDC plug-in installed for your kubectl.

All Kubernetes kubectl commands are routed through the API endpoint created by Omni, and Omni validates access through the configured OIDC provider or other user authorization mechanism. This ensures your Kubernetes cluster is safe - unlike other systems, mere possession of a kubeconfig grants no access - the user also has be valid in the configured authentication system of Omni.

Download the KubeConfig file

Navigate to the clusters page by clicking on the “Clusters” button in the sidebar.

Click on the cluster and then click "Download kubeconfig" from the cluster dashboard on the right. The downloaded file will reflect the name of the cluster.

Install the OIDC plug in

Access the cluster with kubectl

Be sure you use the name of the downloaded kubeconfig file, which will vary with the name of the cluster.

The first time you use the kubectl command to query a cluster, a browser window will open requiring you to authenticate with your identity provider.

If you get a message error: unknown command "oidc-login" for "kubectl" Unable to connect to the server then you need to install the oidc-login plugin as noted above, and ensure it is in your $PATH.

Switching between users when authenticating to the same cluster

To work around that, when you want to switch to another user, you need to clear the authentication cache first. To do this, run one of the following:

After doing this, the next kubectl command you run should trigger the OIDC login flow again, where you can authenticate as the user you need via Switch User option.

OIDC authentication over SSH

If you need to use kubectl, talosctl, or omnictl on a remote host over SSH you may need a way to forward your local client traffic to the remote host where kubectl-oidc_login is installed.

To do that you can tunnel the ports over SSH when you connect to the host. This command will open a tunnel using the default ports oidc-login attempts to use.

You can run this in a separate terminal temporarily while you authenticate your CLI tools.

If you want to have the port forwarding happen automatically every time you connect to the host you should update your ~/.ssh/config file to contain the following lines for the host.

You will also need to disable automatic browser opening because it will likely try to open a browser on the SSH host or fail to open if one is not installed. Do that by adding --skip-open-browser in your $KUBECONFIG file.

Dashboard

Upon logging in you will be presented with the Omni dashboard.

Download the ISO

First, download the ISO from the Omni portal by clicking on the “Download Installation Media” button on the right.

Select the version of Talos for the cluster, then click on the “Options” dropdown menu and search for, or scroll down to, the “ISO” option.

Notice there are two options: one for amd64 and another for arm64. Select the appropriate architecture for the machine you are registering. Now that you have selected the ISO option for the appropriate architecture, click the “Download” button.

Write the ISO to a USB Stick

First, plug the USB drive into your local machine. Now, find the device path for your USB drive and write the ISO to the USB drive.

Boot the Machine

Now that we have our bootable USB drive, plug it into the machine you are registering. Once the machine is booting you will notice logs from Talos Linux on the console stating that it is reachable over an IP address.

Conclusion

Navigate to the “Machines” menu in the sidebar of Omni. You should now see a machine listed.

You now have a bare metal machine registered with Omni and ready to provision.

Dashboard

Upon logging in you will be presented with the Omni dashboard.

Download the Image

Download the Azure image from the Omni portal by clicking on the “Download Installation Media” button. Click on the “Options” dropdown menu and search for the “Azure” option. Notice there are two options: one for amd64 and another for arm64. Select the appropriate architecture for the machine you are registering, then click the “Download” button.

Once downloaded to your local machine, untar with tar -xvf /path/to/image

Upload the Image

Convert the Image

Create an Azure Instance

Conclusion

In the Omni UI, navigate to the “Machines” menu in the sidebar. You should now see the Azure machine that was created listed as an available machine, registered with Omni and ready to provision.

First of all, check the current overall status of the cluster backup subsystem:

omnictl get etcdbackupoverallstatus

If you have freshly created Omni instance, the output will be similar to this:

NAMESPACE   TYPE                      ID                          VERSION   CONFIGURATION NAME   CONFIGURATION ERROR   LAST BACKUP STATUS   LAST BACKUP ERROR   LAST BACKUP TIME   CONFIGURATION ATTEMPT
ephemeral   EtcdBackupOverallStatus   etcdbackup-overall-status   1         s3                   not initialized

The combination of the CONFIGURATION NAME and CONFIGURATION ERROR fields display the current backup store configuration status. Currently, Omni supports two backup stores: local and s3. These are configured during Omni initialization. The output above indicates that the backup store is set to use the s3 store. However, the s3 configuration itself has not yet been added, so the CONFIGURATION ERROR field shows not initialized. The rest of the fields show as empty because no backups have been created yet.

S3 configuration

To use S3 as the backup storage, you will first need to configure the S3 credentials for Omni to use. This can be done by creating an EtcdBackupS3Configs.omni.sidero.dev resource in Omni. Below is an example for Minio S3:

metadata:
  namespace: default
  type: EtcdBackupS3Configs.omni.sidero.dev
  id: etcd-backup-s3-conf
spec:
  bucket: mybucket
  region: us-east-1
  endpoint: http://127.0.0.1:9000
  accesskeyid: access
  secretaccesskey: secret123
  sessiontoken: ""

Let's go through the fields:

• bucket - the name of the S3 bucket for storing backups. This is the only field required in all cases.

• region - the region of the S3 bucket. If not provided, Omni will use the default region.

• endpoint - the S3 endpoint. If not provided, Omni will use the default AWS S3 endpoint.

• accesskeyid and secretaccesskey - the credentials to access the S3 bucket. If not provided, Omni will assume it runs in an EC2 instance with an IAM role that has access to the specified S3 bucket.

• sessiontoken - the session token (if any) for accessing the S3 bucket.

Save it as <file-name>.yaml and apply using omnictl apply -f <file-name>.yaml. During resource creation, Omni will validate the provided credentials by attempting to list the objects in the bucket. It will return an error if the validation fails and will not update the resource.

Let's get our overall status again and check the output:

NAMESPACE   TYPE                      ID                          VERSION   CONFIGURATION NAME   CONFIGURATION ERROR   LAST BACKUP STATUS   LAST BACKUP ERROR   LAST BACKUP TIME   CONFIGURATION ATTEMPT
ephemeral   EtcdBackupOverallStatus   etcdbackup-overall-status   2         s3

Note that the CONFIGURATION ERROR field is now empty, indicating that the provided configuration is valid.

Manual backup

Now, let's create a manual backup. To do that, we need to create a resource:

metadata:
  namespace: ephemeral
  type: EtcdManualBackups.omni.sidero.dev
  id: <your-cluster-name>
spec:
  backupat:
    seconds: <unix-timestamp>
    nanos: 0

The <unix-timestamp> should be no more than one minute in the future or in the past. The easiest way to get the current timestamp is to simply invoke date +%s in your shell. The nanos field should always be 0.

After you save the resource as <file-name>.yaml, apply it using omnictl apply -f <file-name>.yaml. In a few seconds, you can check the status of the backup:

omnictl get etcdbackupstatus -o yaml

This command print per-cluster backup status. The output will be similar to this:

metadata:
  namespace: ephemeral
  type: EtcdBackupStatuses.omni.sidero.dev
  id: <cluster-name>
  version: 1
  owner: EtcdBackupController
  phase: running
spec:
  status: 1
  error: ""
  lastbackuptime:
    seconds: 1702166400
    nanos: 985220192
  lastbackupattempt:
    seconds: 1702166400
    nanos: 985220192

You can also get the overall status of the backup subsystem:

metadata:
  namespace: ephemeral
  type: EtcdBackupOverallStatuses.omni.sidero.dev
  id: etcdbackup-overall-status
  version: 3
  owner: EtcdBackupOverallStatusController
  phase: running
spec:
  configurationname: s3
  configurationerror: ""
  lastbackupstatus:
    status: 1
    error: ""
    lastbackuptime:
      seconds: 1702166400
      nanos: 985220192
    lastbackupattempt:
      seconds: 1702166400
      nanos: 985220192

Automatic backup

Omni also supports automatic backups. You can enable this feature by directly editing the cluster resource Clusters.omni.sidero.dev or by using cluster templates. Let's explore how we can do this in both ways.

Cluster templates

Enabling automatic backups using cluster templates is quite straightforward. First, you'll need a template that resembles the following:

kind: Cluster
name: talos-default
kubernetes:
  version: v1.28.2
talos:
  version: v1.5.5
features:
  backupConfiguration:
    interval: 1h
---
kind: ControlPlane
machines:
  - 1dd4397b-37f1-4196-9c37-becef670b64a
---
kind: Workers
machines:
  - 0d1f01c3-0a8a-4560-8745-bb792e3dfaad
  - a0f29661-cd2d-4e25-a6c9-da5ca4c48d58

This is the minimal example of a cluster template for a cluster with a single-node control plane and two worker nodes. Your machine UUIDs will be different, and the Kubernetes and Talos versions will probably also differ. You will need both of these, as well as the cluster name, in your cluster template. To obtain these, refer to the clustermachinestatus and cluster resources.

In this example, we are going to set the backup interval for the cluster to one hour. Save this template as <file-name>.yaml. Before applying this change, we want to ensure that no automatic backup is enabled for this cluster. To do that, let's run the following command:

omnictl cluster template -f <file-name>.yaml diff

The Omni response will resemble the following:

--- Clusters.omni.sidero.dev(default/talos-default)
+++ Clusters.omni.sidero.dev(default/talos-default)
@@ -19,4 +19,7 @@
   features:
     enableworkloadproxy: false
     diskencryption: false
-  backupconfiguration: null
+  backupconfiguration:
+    interval:
+      seconds: 3600
+      nanos: 0

Now that we have verified that Omni does not already have an automatic backup enabled, we will apply the change:

omnictl cluster template -f <file-name>.yaml sync

If you didn't have any backups previously, Omni will not wait for an hour and will immediately create a fresh backup. You can verify this by running the following command:

omnictl get etcdbackup --selector omni.sidero.dev/cluster=talos-default

Keep in mind that to obtain the backup status, you will need to use the label selector omni.sidero.dev/cluster along with your cluster name. In this example it is talos-default.

NAMESPACE   TYPE         ID                         VERSION     CREATED AT
external    EtcdBackup   talos-default-1702166400   undefined   {"nanos":0,"seconds":1702166400}

Cluster resource

Another way to enable automatic backups is by directly editing the cluster resource. To do this, first, you'll need to retrieve the cluster resource from the Omni:

omnictl get cluster talos-default -o yaml

metadata:
  namespace: default
  type: Clusters.omni.sidero.dev
  id: talos-default
  finalizers:
    - KubernetesUpgradeStatusController
    - TalosUpgradeStatusController
    - SecretsController
    - ClusterController
spec:
  installimage: ""
  kubernetesversion: 1.28.2
  talosversion: 1.5.5
  features:
    enableworkloadproxy: false
    diskencryption: false

Add fields related to the backup configuration while preserving the existing fields:

metadata:
  namespace: default
  type: Clusters.omni.sidero.dev
  id: talos-default
  finalizers:
    - KubernetesUpgradeStatusController
    - TalosUpgradeStatusController
    - SecretsController
    - ClusterController
spec:
  installimage: ""
  kubernetesversion: 1.28.2
  talosversion: 1.5.5
  features:
    enableworkloadproxy: false
    diskencryption: false
  backupconfiguration:
    interval:
      seconds: 3600
      nanos: 0

Save it to the file and apply using omnictl apply -f <file-name>.yaml. You will get the output similar to the one above for the cluster template.

Dashboard

Upon logging in you will be presented with the Omni dashboard.

Download the Image

First, download the GCP image from the Omni portal by clicking on the “Download Installation Media” button. Now, click on the “Options” dropdown menu and search for the “GCP” option. Notice there are two options: one for amd64 and another for arm64. Select the appropriate option for the machine you are registering. Now that you have selected the GCP option for the appropriate architecture, click the “Download” button.

Upload the Image

In the Google Cloud console, navigate to Buckets under the Cloud Storage menu, and create a new bucket with the default. Click on the bucket in the Google Cloud console, click Upload Files, and select the image download from the Omni console.

Convert the Image

In the Google Cloud console select Images under the Compute Engine menu, and then Create Image. Name your image (e.g. Omni-talos-1.7.6), then select the Source as Cloud Storage File. Click Browse in the Cloud Storage File field and navigate to the bucket you created. Select the image you uploaded. Leave the rest of the options at their default and click Create at the bottom.

Create a GCP Instance

In Google Cloud console select VM instances under the Compute Engine menu. Now select Create Instance. Name your instance, and select a region and zone. Under “Machine Configuration”, ensure your instance has at least 4GB of memory. In the Boot Disk section, select Change and then select Custom Images. Select the image created in the previous steps. Now, click Create at the bottom to create your instance.

Conclusion

Navigate to the “Machines” menu in the sidebar. You should now see a machine listed.

You now have a GCP machine registered with Omni and ready to provision.

Omni deployment configuration

--name=$OMNI_NAME
--private-key-source=file:///omni.asc
--advertised-api-url=https://$OMNI_DOMAIN/
--bind-addr=127.0.0.1:8080
--machine-api-bind-addr=127.0.0.1:8090
--siderolink-api-advertised-url=https://api.$OMNI_DOMAIN:443
--k8s-proxy-bind-addr=127.0.0.1:8100
--advertised-kubernetes-proxy-url=https://kube.$OMNI_DOMAIN/
--account-id=$OMNI_UUID
--siderolink-use-grpc-tunnel=true
## Also add the authentication flags according to your setup

Certificates

You can use acme or certbot to generate certificates for your domain. In the following nginx config, the are stored in /var/lib/acme/omni/ and /var/lib/acme/omni_api/ and /var/lib/acme/omni_kube/. Make sure to change the paths to your own or to output the certificates to those paths.

Nginx configuration

Use the following configuration to expose omni with nginx. Make sure to change the domain name ($OMNI_DOMAIN) to your own domain and to update the certificate paths if applicable.

http {
	proxy_redirect          off;
	proxy_connect_timeout   60s;
	proxy_send_timeout      60s;
	proxy_read_timeout      60s;
	proxy_http_version      1.1;
	# $connection_upgrade is used for websocket proxying
	map $http_upgrade $connection_upgrade {
		default upgrade;
		''      close;
	}
    server {
		listen 0.0.0.0:80 ;
		listen [::0]:80 ;
		server_name $OMNI_DOMAIN ;
		location / {
			return 301 https://$host$request_uri;
		}
	}
	server {
		listen 0.0.0.0:443 http2 ssl ;
		listen [::0]:443 http2 ssl ;
		server_name $OMNI_DOMAIN ;
		ssl_certificate /var/lib/acme/omni/fullchain.pem;
		ssl_certificate_key /var/lib/acme/omni/key.pem;
		ssl_trusted_certificate /var/lib/acme/omni/chain.pem;
		location / {
			proxy_pass http://127.0.0.1:8080;
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection $connection_upgrade;
			grpc_pass grpc://127.0.0.1:8080;
		}
	}
	server {
		listen 0.0.0.0:443 http2 ssl ;
		listen [::0]:443 http2 ssl ;
		server_name api.$OMNI_DOMAIN ;
		ssl_certificate /var/lib/acme/omni_api/fullchain.pem;
		ssl_certificate_key /var/lib/acme/omni_api/key.pem;
		ssl_trusted_certificate /var/lib/acme/omni_api/chain.pem;
		location / {
			grpc_pass grpc://127.0.0.1:8090;
		}
	}
	server {
		listen 0.0.0.0:443 http2 ssl ;
		listen [::0]:443 http2 ssl ;
		server_name kube.$OMNI_DOMAIN ;
		ssl_certificate /var/lib/acme/omni_kube/fullchain.pem;
		ssl_certificate_key /var/lib/acme/omni_kube/key.pem;
		ssl_trusted_certificate /var/lib/acme/omni_kube/chain.pem;
		location / {
			proxy_pass http://127.0.0.1:8100;
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection $connection_upgrade;
		}
	}
}

How to use

The omni instance will be available at https://$OMNI_DOMAIN/, the API at https://api.$OMNI_DOMAIN/ and the kubernetes proxy at https://kube.$OMNI_DOMAIN/.

This guide shows you how to back up the database of an on-prem Omni instance.

Omni uses etcd as its database.

There are 2 operating modes for etcd: embedded and external.

When Omni is run with --etcd-embedded=true flag, it will configure the embedded etcd server to listen the addresses specified by the --etcd-endpoints flag (http://localhost:2379 by default).

etcdctl --endpoints http://localhost:2379 snapshot save snapshot.db

The command will save the snapshot of the database to the snapshot.db file.

It is recommended to periodically (e.g. with a cron job) take snapshots and store them in a safe location, like an S3 bucket.

This guide shows you how to deploy Omni on-prem. This guide assumes that Omni will be deployed on an Ubuntu machine. Small differences should be expected when using a different OS.

For SAML integration sections, this guide assumes Azure AD will be the provider for SAML.

Prerequisites

There are several prerequisites for deploying Omni on-prem. We will assume you have an Ubuntu machine available. Any distribution with Docker should work.

Install Docker

curl -L https://get.docker.io | sh

Generate Certs

# Install certbot
$ sudo snap install --classic certbot

# Allow for root access
$ sudo snap set certbot trust-plugin-with-root=ok

# Install DNS provider
$ snap install certbot-dns-<provider>

# Create creds file with API tokens
$ echo '<creds example' > creds.ini

# Create certs for desired domain
$ certbot certonly --dns-<provider> -d <domain name for onprem omni>

Configure Authentication

Auth0

On the account level, configure "Authentication - Social" to allow GitHub and Google login.

Create an Auth0 application of the type "single page web application".

Configure the Auth0 application with the following:

• Allowed callback URLs: https://<domain name for onprem omni>

• Allowed web origins: https://<domain name for onprem omni>

• Allowed logout URLs: https://<domain name for onprem omni>

Disable username/password auth on "Authentication - Database - Applications" tab.

Enable GitHub and Google login on the "Authentication - Social" tab.

Enable email access in the GitHub settings.

Take note of the following information from the Auth0 application:

• Domain

• Client ID

SAML Identity Providers

Other identity providers also work with Omni. Configuring these should be similar to Auth0.

Create Etcd Encryption Key

Generate a GPG key:

gpg --quick-generate-key "Omni (Used for etcd data encryption) how-to-guide@siderolabs.com" rsa4096 cert never

Find the fingerprint of the generated key:

gpg --list-secret-keys

Using the fingerprint, add an encryption subkey and export:

gpg --quick-add-key <fingerprint> rsa4096 encr never
gpg --export-secret-key --armor how-to-guide@siderolabs.com > omni.asc

Note: Do not add passphrases to keys during creation.

Deploy Omni

There are two easy ways to run Omni: docker-compose and a simple docker run. We recommend using docker-compose, but both are detailed in separate tabs below.

export OMNI_VERSION=0.41.0
OMNI_ACCOUNT_UUID=$(uuidgen)
OMNI_DOMAIN_NAME=omni.siderolabs.com
OMNI_WG_IP=10.10.1.100
OMNI_ADMIN_EMAIL=omni@siderolabs.com
AUTH0_CLIENT_ID=xxxyyyzzz
AUTH0_DOMAIN=dev-aaabbbccc.us.auth0.com

curl https://raw.githubusercontent.com/siderolabs/omni/v${OMNI_VERSION}/deploy/env.template \
  | envsubst > omni.env

curl https://raw.githubusercontent.com/siderolabs/omni/v${OMNI_VERSION}/deploy/compose.yaml -o compose.yaml

docker compose --env-file omni.env up -d

docker run \
  --net=host \
  --cap-add=NET_ADMIN \
  --device /dev/net/tun \
  -v $PWD/etcd:/_out/etcd \
  -v <path to TLS certificate>:/tls.crt \
  -v <path to TLS key>:/tls.key \
  -v $PWD/omni.asc:/omni.asc \
  ghcr.io/siderolabs/omni:<tag> \
    --account-id=$(uuidgen) \
    --name=onprem-omni \
    --cert=/tls.crt \
    --key=/tls.key \
    --siderolink-api-cert=/tls.crt \
    --siderolink-api-key=/tls.key \
    --private-key-source=file:///omni.asc \
    --event-sink-port=8091 \
    --bind-addr=0.0.0.0:443 \
    --siderolink-api-bind-addr=0.0.0.0:8090 \
    --k8s-proxy-bind-addr=0.0.0.0:8100 \
    --advertised-api-url=https://<domain name for onprem omni>/ \
    --siderolink-api-advertised-url=https://<domain name for onprem omni>:8090/ \
    --siderolink-wireguard-advertised-addr=<ip address of the host running Omni>:50180 \
    --advertised-kubernetes-proxy-url=https://<domain name for onprem omni>:8100/ \
    --auth-auth0-enabled=true \
    --auth-auth0-domain=<Auth0 domain> \
    --auth-auth0-client-id=<Auth0 client ID> \
    --initial-users=<email address>

docker run \
  --net=host \
  --cap-add=NET_ADMIN \
  --device /dev/net/tun \
  -v $PWD/etcd:/_out/etcd \
  -v <path to full chain TLS certificate>:/tls.crt \
  -v <path to TLS key>:/tls.key \
  -v $PWD/omni.asc:/omni.asc \
  ghcr.io/siderolabs/omni:<tag> \
    --account-id=$(uuidgen) \
    --name=onprem-omni \
    --cert=/tls.crt \
    --key=/tls.key \
    --siderolink-api-cert=/tls.crt \
    --siderolink-api-key=/tls.key \
    --private-key-source=file:///omni.asc \
    --event-sink-port=8091 \
    --bind-addr=0.0.0.0:443 \
    --siderolink-api-bind-addr=0.0.0.0:8090 \
    --k8s-proxy-bind-addr=0.0.0.0:8100 \
    --advertised-api-url=https://<domain name for onprem omni>/ \
    --siderolink-api-advertised-url=https://<domain name for onprem omni>:8090/ \
    --siderolink-wireguard-advertised-addr=<ip address of the host running Omni>:50180 \
    --advertised-kubernetes-proxy-url=https://<domain name for onprem omni>:8100/ \
    --auth-saml-enabled=true \
    --auth-saml-url=<app federation metadata url copied during Azure AD setup>

Creating the Kubernetes Service Account Kubeconfig

To create a service account kubeconfig, run the following command:

omnictl kubeconfig --service-account -c <cluster> --user <username> <path to kubeconfig>

This command will create a service account token with the given username and obtain a kubeconfig file for the given cluster and username.

You can now use kubectl with the generated kubeconfig.

This guide shows you how to create a cluster consisting of any combination of bare metal, cloud virtual machines, on-premise virtual machines, or SBCs, using KubeSpan. KubeSpan is a feature of Talos Linux that provides full node-to-node network encryption with WireGuard, and enables Kubernetes to operate securely even when machines in the cluster are on different networks.

Refer to the general guide on creating a cluster to get started. To create a hybrid cluster, navigate to the cluster, then apply the following cluster patch by clicking on "Config Patches", and create a new patch with the target of "Cluster":

All machines in this cluster will have this patch applied to them, and use WireGuard encryption for all node-to-node traffic.

Creating the Service Account

To create an Omni service account, use the following command:

The output of this command will print OMNI_ENDPOINT and OMNI_SERVICE_ACCOUNT_KEY.

Export these variables with the printed values:

You can now use omnictl with the generated service account.