You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

48 lines
2.2 KiB

  1. Large deployments of K8s
  2. ========================
  3. For a large scaled deployments, consider the following configuration changes:
  4. * Tune [ansible settings]
  5. (http://docs.ansible.com/ansible/intro_configuration.html)
  6. for `forks` and `timeout` vars to fit large numbers of nodes being deployed.
  7. * Override containers' `foo_image_repo` vars to point to intranet registry.
  8. * Override the ``download_run_once: true`` and/or ``download_localhost: true``.
  9. See download modes for details.
  10. * Adjust the `retry_stagger` global var as appropriate. It should provide sane
  11. load on a delegate (the first K8s master node) then retrying failed
  12. push or download operations.
  13. * Tune parameters for DNS related applications (dnsmasq daemon set, kubedns
  14. replication controller). Those are ``dns_replicas``, ``dns_cpu_limit``,
  15. ``dns_cpu_requests``, ``dns_memory_limit``, ``dns_memory_requests``.
  16. Please note that limits must always be greater than or equal to requests.
  17. * Tune CPU/memory limits and requests. Those are located in roles' defaults
  18. and named like ``foo_memory_limit``, ``foo_memory_requests`` and
  19. ``foo_cpu_limit``, ``foo_cpu_requests``. Note that 'Mi' memory units for K8s
  20. will be submitted as 'M', if applied for ``docker run``, and cpu K8s units
  21. will end up with the 'm' skipped for docker as well. This is required as
  22. docker does not understand k8s units well.
  23. * Tune ``kubelet_status_update_frequency`` to increase reliability of kubelet.
  24. ``kube_controller_node_monitor_grace_period``,
  25. ``kube_controller_node_monitor_period``,
  26. ``kube_controller_pod_eviction_timeout`` for better Kubernetes reliability.
  27. Check out [Kubernetes Reliability](kubernetes-reliability.md)
  28. * Add calico-rr nodes if you are deploying with Calico or Canal. Nodes recover
  29. from host/network interruption much quicker with calico-rr. Note that
  30. calico-rr role must be on a host without kube-master or kube-node role (but
  31. etcd role is okay).
  32. * Check out the
  33. [Inventory](getting-started.md#building-your-own-inventory)
  34. section of the Getting started guide for tips on creating a large scale
  35. Ansible inventory.
  36. For example, when deploying 200 nodes, you may want to run ansible with
  37. ``--forks=50``, ``--timeout=600`` and define the ``retry_stagger: 60``.