Getting Started with Omni
Setting up a Talos Linux cluster with Omni.
Last updated
Setting up a Talos Linux cluster with Omni.
Last updated
In this Getting Started guide we will create a high availability Kubernetes cluster with Talos Linux managed by Omni. This guide will use UTM/QEMU, but the same process will work with bare metal machines, cloud instances, and edge devices.
If you would like to watch a quick video of the process you can follow along here.
If your machines have outgoing internet access, you are all set. Machines should have access to the Wireguard Endpoint shown on the Omni Home panel, which lists the IP address and port (e.g. 199.99.99.100:10001) that machines will connect to. Machines need to be able to reach that address both on the UDP port specified, and on TCP port 443.
The simplest way to experience Omni is to create some virtual machines. We suggest any virtualization platform that can boot off an ISO (UTM, ProxMox, VMware Fusion, etc) although any cloud platform can also be used with minor adjustments. Bare metal can boot from a physical CD, USB drive, virtual media, or PXE.
talosctltalosctl is the command line tool for managing Talos Linux via the management API, but when machines connect to Omni it is not required. Instead cluster management is done via the Omni UI or omnictl. We still recommend installing talosctl to investigate the state of the nodes and explore functionality.
Download talosctl, kubectl, kubectl-oidc_login, and omnictl (macOS and Linux):
For manual and Windows installation please refer to the alternate installation methods in the Talos documentation.
Make sure you download the talosconfig and omniconfig files from the Omni web interface and place them in the default locations ~/.talos/config and ~/.config/omni/config/ respectively.
Omni is a BYO Machine platform - all you need to do is boot your machines (virtual or physical) off an Omni image. To download the installation media, go to the Home Screen in Omni, and select "Download Installation Media" from the right hand side. Select the appropriate media and platform type.
Images exist for many platforms, but you will have to follow the specific installation instructions for that platform (which often involve copying the image to S3 type storage, creating a machine image from it, etc.)
Create at least 1 virtual machine with 2GB of memory (4GB or more is recommended) using your Hypervisor. Have each virtual machine boot off the ISO image you just downloaded.
After a few seconds, the machines should show in the Machines panel of Omni, with the available tag. They will also have tags showing their architecture, memory, cores and other information.
Click "Clusters" on the left navigation panel, then "Create Cluster" in the top right. You can give your cluster a name, select the version of Talos Linux to install, and the version of Kubernetes. You can also specify any patches that should be applied to your cluster, but in most cases these are not needed to get started.
In the section headed "Available Machines", select at least one machine to be the control plane, by clicking CP. You will want an odd number of control plane nodes (e.g. 1, 3, 5). Select one machine to be a worker, by clicking W0 next to the machine.
Then click Create Cluster. Your cluster is now being created, and you will be taken to the Cluster Overview page. From this page you can download the kubeconfig and talosconfig files for your cluster, by clicking the buttons on the right hand side.
Please note that because Omni manages the state of the Talos nodes, and protects the security of the Kubernetes and Talos credentials. Because of this, some
talosctlcommands (such astalosctl reset) will returnPermissionDeniedon Omni managed clusters - such operations must be done through the Omni UI or API calls.
You can query your Kubernetes cluster using normal Kubernetes operations:
Note: you will have to change the referenced kubeconfig file depending on the name of the cluster you created.
The first time you use the kubectl command to query a cluster, a browser window will open requiring you to authenticate with your identity provider (Google or GitHub most commonly.) If you get a message error: unknown command "oidc-login" for "kubectl" Unable to connect to the server: then you need to install the oidc-login plugin as noted above.
You can explore Talos API commands. The first time you access the Talos API, a browser window will start to authenticate your request. The downloaded talosconfig file for the cluster includes the Omni endpoint, so you do not need to specify endpoints, just nodes.
You will need to change the name of the
talosconfigfile, if you changed the cluster name from the default; and also use the actual IP or name of the nodes you created (which are shown in Omni) in place of thenodeIP.
Note that because Omni manages the state of the Talos nodes, and protects the security of the Kubernetes and Talos credentials, some
talosctlcommands (such astalosctl reset) will returnPermissionDeniedon Omni managed clusters - such operations must be done through the Omni UI or API calls.
Now you have a complete cluster, with a high-availability Kubernetes API endpoint running on the Omni infrastructure, where all authentication is tied in to your enterprise identity provider. It's a good time to explore all that Omni can offer, including other areas of the UI such as:
etcd backup and restores
simple cluster upgrades of Kubernetes and Operating System
proxying of workload HTTP access
simple scaling up and down of clusters
the concept of Machine Sets, that let you manage your infrastructure by classes
If you want to declaratively manage your clusters and infrastructure declaratively, as code, check out Cluster Templates.
When you are all done, you can remove the cluster by clicking "Destroy Cluster", in the bottom right of the Cluster Overview panel. This will wipe the machines and return them to the Available state.
We have an example of a managed cluster complete with a monitoring stack and application management. It can be found in our community contrib repo.
The contrib example includes:
Cilium for cluster networking
Hubble for network observability
ArgoCD for application management
Rook/Ceph for persistent storage
Prometheus for metrics collection and alerting
Grafana for metrics visualization
You will need to copy the contents of the omni directory to a git repository that can be accessed by the cluster you create. Update the ArgoCD ApplicationSet template to reference your new git repo, and regenerate the ArgoCD bootstrap patch.
With these changes made you should commit the new values and push them to the git repo.
Next you should register your machines with Omni (see guides for AWS, GCP, Azure, Hetzner, and bare metal) and create machine classes to match your hardware. By default, the example cluster template is configured to use 3 instances of machine classes named omni-contrib-controlplane, and all instances that match a machine class called omni-contrib-workers. You can modify these settings in the cluster-template.yaml, but keep in mind that for Rook/Ceph to work you will need to use at least 3 instances with additional block devices for storage.
Once machines are registered you can create the cluster using the cluster template in the infra directory.
This should create the cluster as described, bootstrap ArgoCD, and begin installing applications from your repo. Depending on your infrastructure, it should take 5-10 mins for the cluster to come fully online with all applications working and healthy. Monitoring can be viewed directly from Omni using the workload proxy feature, with links to Grafana and Hubble found on the left-hand side of the Omni UI.