to create or modify an Ansible inventory. Currently, it is limited in
to create or modify an Ansible inventory. Currently, it is limited in
functionality and is only used for configuring a basic Kubespray cluster inventory, but it does
functionality and is only used for configuring a basic Kubespray cluster inventory, but it does
support creating inventory file for large clusters as well. It now supports
support creating inventory file for large clusters as well. It now supports
separated ETCD and Kubernetes master roles from node role if the size exceeds a
separated ETCD and Kubernetes control plane roles from node role if the size exceeds a
certain threshold. Run `python3 contrib/inventory_builder/inventory.py help` for more information.
certain threshold. Run `python3 contrib/inventory_builder/inventory.py help` for more information.
Example inventory generator usage:
Example inventory generator usage:
@ -40,7 +40,7 @@ See more details in the [ansible guide](/docs/ansible.md).
### Adding nodes
### Adding nodes
You may want to add worker, master or etcd nodes to your existing cluster. This can be done by re-running the `cluster.yml` playbook, or you can target the bare minimum needed to get kubelet installed on the worker and talking to your masters. This is especially helpful when doing something like autoscaling your clusters.
You may want to add worker, control plane or etcd nodes to your existing cluster. This can be done by re-running the `cluster.yml` playbook, or you can target the bare minimum needed to get kubelet installed on the worker and talking to your control planes. This is especially helpful when doing something like autoscaling your clusters.
- Add the new worker node to your inventory in the appropriate group (or utilize a [dynamic inventory](https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html)).
- Add the new worker node to your inventory in the appropriate group (or utilize a [dynamic inventory](https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html)).
- Run the ansible-playbook command, substituting `cluster.yml` for `scale.yml`:
- Run the ansible-playbook command, substituting `cluster.yml` for `scale.yml`:
You may want to remove **master**, **worker**, or **etcd** nodes from your
You may want to remove **control plane**, **worker**, or **etcd** nodes from your
existing cluster. This can be done by re-running the `remove-node.yml`
existing cluster. This can be done by re-running the `remove-node.yml`
playbook. First, all specified nodes will be drained, then stop some
playbook. First, all specified nodes will be drained, then stop some
kubernetes services and delete some certificates,
kubernetes services and delete some certificates,
@ -108,11 +108,11 @@ Accessing through Ingress is highly recommended. For proxy access, please note t
For token authentication, guide to create Service Account is provided in [dashboard sample user](https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md) doc. Still take care of default namespace.
For token authentication, guide to create Service Account is provided in [dashboard sample user](https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md) doc. Still take care of default namespace.
Access can also by achieved via ssh tunnel on a master :
Access can also by achieved via ssh tunnel on a control plane :
```bash
```bash
# localhost:8081 will be sent to master-1's own localhost:8081
ssh -L8001:localhost:8001 user@master-1
# localhost:8081 will be sent to control-plane-1's own localhost:8081
@ -6,9 +6,9 @@ Modified from [comments in #3471](https://github.com/kubernetes-sigs/kubespray/i
Currently you can't remove the first node in your kube_control_plane and etcd-master list. If you still want to remove this node you have to:
Currently you can't remove the first node in your kube_control_plane and etcd-master list. If you still want to remove this node you have to:
### 1) Change order of current masters
### 1) Change order of current control planes
Modify the order of your master list by pushing your first entry to any other position. E.g. if you want to remove `node-1` of the following example:
Modify the order of your control plane list by pushing your first entry to any other position. E.g. if you want to remove `node-1` of the following example:
```yaml
```yaml
children:
children:
@ -71,13 +71,13 @@ Before using `--limit` run playbook `facts.yml` without the limit to refresh fac
With the old node still in the inventory, run `remove-node.yml`. You need to pass `-e node=NODE_NAME` to the playbook to limit the execution to the node being removed.
With the old node still in the inventory, run `remove-node.yml`. You need to pass `-e node=NODE_NAME` to the playbook to limit the execution to the node being removed.
If the node you want to remove is not online, you should add `reset_nodes=false` to your extra-vars: `-e node=NODE_NAME -e reset_nodes=false`.
If the node you want to remove is not online, you should add `reset_nodes=false` to your extra-vars: `-e node=NODE_NAME -e reset_nodes=false`.
Use this flag even when you remove other types of nodes like a master or etcd nodes.
Use this flag even when you remove other types of nodes like a control plane or etcd nodes.
### 4) Remove the node from the inventory
### 4) Remove the node from the inventory
That's it.
That's it.
## Adding/replacing a master node
## Adding/replacing a control plane node
### 1) Run `cluster.yml`
### 1) Run `cluster.yml`
@ -92,7 +92,7 @@ In all hosts, restart nginx-proxy pod. This pod is a local proxy for the apiserv
With the old node still in the inventory, run `remove-node.yml`. You need to pass `-e node=NODE_NAME` to the playbook to limit the execution to the node being removed.
With the old node still in the inventory, run `remove-node.yml`. You need to pass `-e node=NODE_NAME` to the playbook to limit the execution to the node being removed.
If the node you want to remove is not online, you should add `reset_nodes=false` to your extra-vars.
If the node you want to remove is not online, you should add `reset_nodes=false` to your extra-vars.
@ -104,7 +104,7 @@ You need to make sure there are always an odd number of etcd nodes in the cluste
### 1) Add the new node running cluster.yml
### 1) Add the new node running cluster.yml
Update the inventory and run `cluster.yml` passing `--limit=etcd,kube_control_plane -e ignore_assert_errors=yes`.
Update the inventory and run `cluster.yml` passing `--limit=etcd,kube_control_plane -e ignore_assert_errors=yes`.
If the node you want to add as an etcd node is already a worker or master node in your cluster, you have to remove him first using `remove-node.yml`.
If the node you want to add as an etcd node is already a worker or control plane node in your cluster, you have to remove him first using `remove-node.yml`.
Run `upgrade-cluster.yml` also passing `--limit=etcd,kube_control_plane -e ignore_assert_errors=yes`. This is necessary to update all etcd configuration in the cluster.
Run `upgrade-cluster.yml` also passing `--limit=etcd,kube_control_plane -e ignore_assert_errors=yes`. This is necessary to update all etcd configuration in the cluster.
@ -117,7 +117,7 @@ Otherwise the etcd cluster might still be processing the first join and fail on
### 2) Add the new node to apiserver config
### 2) Add the new node to apiserver config
In every master node, edit `/etc/kubernetes/manifests/kube-apiserver.yaml`. Make sure the new etcd nodes are present in the apiserver command line parameter `--etcd-servers=...`.
In every control plane node, edit `/etc/kubernetes/manifests/kube-apiserver.yaml`. Make sure the new etcd nodes are present in the apiserver command line parameter `--etcd-servers=...`.
## Removing an etcd node
## Removing an etcd node
@ -136,7 +136,7 @@ Run `cluster.yml` to regenerate the configuration files on all remaining nodes.
### 4) Remove the old etcd node from apiserver config
### 4) Remove the old etcd node from apiserver config
In every master node, edit `/etc/kubernetes/manifests/kube-apiserver.yaml`. Make sure only active etcd nodes are still present in the apiserver command line parameter `--etcd-servers=...`.
In every control plane node, edit `/etc/kubernetes/manifests/kube-apiserver.yaml`. Make sure only active etcd nodes are still present in the apiserver command line parameter `--etcd-servers=...`.
@ -20,7 +20,7 @@ __Note that you need at least one functional node to be able to recover using th
## Runbook
## Runbook
* Move any broken etcd nodes into the "broken\_etcd" group, make sure the "etcd\_member\_name" variable is set.
* Move any broken etcd nodes into the "broken\_etcd" group, make sure the "etcd\_member\_name" variable is set.
* Move any broken master nodes into the "broken\_kube\_control\_plane" group.
* Move any broken control plane nodes into the "broken\_kube\_control\_plane" group.
Then run the playbook with ```--limit etcd,kube_control_plane``` and increase the number of ETCD retries by setting ```-e etcd_retries=10``` or something even larger. The amount of retries required is difficult to predict.
Then run the playbook with ```--limit etcd,kube_control_plane``` and increase the number of ETCD retries by setting ```-e etcd_retries=10``` or something even larger. The amount of retries required is difficult to predict.
For all kube components, custom flags can be passed in. This allows for edge cases where users need changes to the default deployment that may not be applicable to all deployments.
For all kube components, custom flags can be passed in. This allows for edge cases where users need changes to the default deployment that may not be applicable to all deployments.
Extra flags for the kubelet can be specified using these variables,
Extra flags for the kubelet can be specified using these variables,
in the form of dicts of key-value pairs of configuration parameters that will be inserted into the kubelet YAML config file. The `kubelet_node_config_extra_args` apply kubelet settings only to nodes and not masters. Example:
in the form of dicts of key-value pairs of configuration parameters that will be inserted into the kubelet YAML config file. The `kubelet_node_config_extra_args` apply kubelet settings only to nodes and not control planes. Example:
```yml
```yml
kubelet_config_extra_args:
kubelet_config_extra_args:
@ -202,7 +202,7 @@ Previously, the same parameters could be passed as flags to kubelet binary with
* *kubelet_custom_flags*
* *kubelet_custom_flags*
* *kubelet_node_custom_flags*
* *kubelet_node_custom_flags*
The `kubelet_node_custom_flags` apply kubelet settings only to nodes and not masters. Example:
The `kubelet_node_custom_flags` apply kubelet settings only to nodes and not control planes. Example: