Smaine Kahlouch
8 years ago
committed by
GitHub
GitHub
10 changed files with 370 additions and 25 deletions
Split View
Diff Options
-
25README.md
-
1README.md
-
50docs/ansible.md
-
39docs/calico.md
-
22docs/cloud.md
-
24docs/coreos.md
-
75docs/documentation.md
-
51docs/flannel.md
-
19docs/getting-started.md
-
48docs/openstack.md
-
41docs/vagrant.md
@ -1,25 +0,0 @@ |
|||
|
|||
|
|||
 |
|||
|
|||
##Deploy a production ready kubernetes cluster |
|||
|
|||
- Can be deployed on **AWS, GCE, OpenStack or Baremetal** |
|||
- **High available** cluster |
|||
- **Composable** (Choice of the network plugin for instance) |
|||
- Support most popular **Linux distributions** |
|||
- **Continuous integration tests** |
|||
|
|||
|
|||
To deploy the cluster you can use : |
|||
|
|||
* [**kargo-cli**](https://github.com/kubespray/kargo-cli) |
|||
* **vagrant** by simply running `vagrant up` |
|||
* **Ansible** usual commands |
|||
|
|||
A complete **documentation** can be found [**here**](https://docs.kubespray.io) |
|||
|
|||
If you have questions, you can [invite yourself](https://slack.kubespray.io/) to **chat** with us on Slack! [](https://kubespray.slack.com) |
|||
|
|||
[](https://travis-ci.org/kubespray/kargo) </br> |
|||
CI tests sponsored by Google (GCE), and [teuto.net](https://teuto.net/) for OpenStack. |
|||
@ -0,0 +1 @@ |
|||
docs/documentation.md |
|||
@ -0,0 +1,50 @@ |
|||
Ansible variables |
|||
=============== |
|||
|
|||
|
|||
Inventory |
|||
------------- |
|||
The inventory is composed of 3 groups: |
|||
|
|||
* **kube-node** : list of kubernetes nodes where the pods will run. |
|||
* **kube-master** : list of servers where kubernetes master components (apiserver, scheduler, controller) will run. |
|||
Note: if you want the server to act both as master and node the server must be defined on both groups _kube-master_ and _kube-node_ |
|||
* **etcd**: list of server to compose the etcd server. you should have at least 3 servers for failover purposes. |
|||
|
|||
Below is a complete inventory example: |
|||
|
|||
``` |
|||
## Configure 'ip' variable to bind kubernetes services on a |
|||
## different ip than the default iface |
|||
node1 ansible_ssh_host=95.54.0.12 # ip=10.3.0.1 |
|||
node2 ansible_ssh_host=95.54.0.13 # ip=10.3.0.2 |
|||
node3 ansible_ssh_host=95.54.0.14 # ip=10.3.0.3 |
|||
node4 ansible_ssh_host=95.54.0.15 # ip=10.3.0.4 |
|||
node5 ansible_ssh_host=95.54.0.16 # ip=10.3.0.5 |
|||
node6 ansible_ssh_host=95.54.0.17 # ip=10.3.0.6 |
|||
|
|||
[kube-master] |
|||
node1 |
|||
node2 |
|||
|
|||
[etcd] |
|||
node1 |
|||
node2 |
|||
node3 |
|||
|
|||
[kube-node] |
|||
node2 |
|||
node3 |
|||
node4 |
|||
node5 |
|||
node6 |
|||
|
|||
[k8s-cluster:children] |
|||
kube-node |
|||
kube-master |
|||
etcd |
|||
``` |
|||
|
|||
Group vars |
|||
-------------- |
|||
The main variables to change are located in the directory ```inventory/group_vars/all.yml```. |
|||
@ -0,0 +1,39 @@ |
|||
Calico |
|||
=========== |
|||
|
|||
Check if the calico-node container is running |
|||
|
|||
``` |
|||
docker ps | grep calico |
|||
``` |
|||
|
|||
The **calicoctl** command allows to check the status of the network workloads. |
|||
* Check the status of Calico nodes |
|||
|
|||
``` |
|||
calicoctl status |
|||
``` |
|||
|
|||
* Show the configured network subnet for containers |
|||
|
|||
``` |
|||
calicoctl pool show |
|||
``` |
|||
|
|||
* Show the workloads (ip addresses of containers and their located) |
|||
|
|||
``` |
|||
calicoctl endpoint show --detail |
|||
``` |
|||
|
|||
##### Optionnal : BGP Peering with border routers |
|||
|
|||
In some cases you may want to route the pods subnet and so NAT is not needed on the nodes. |
|||
For instance if you have a cluster spread on different locations and you want your pods to talk each other no matter where they are located. |
|||
The following variables need to be set: |
|||
`peer_with_router` to enable the peering with the datacenter's border router (default value: false). |
|||
you'll need to edit the inventory and add a and a hostvar `local_as` by node. |
|||
|
|||
``` |
|||
node1 ansible_ssh_host=95.54.0.12 local_as=xxxxxx |
|||
``` |
|||
@ -0,0 +1,22 @@ |
|||
Cloud providers |
|||
============== |
|||
|
|||
#### Provisioning |
|||
|
|||
You can use kargo-cli to start new instances on cloud providers |
|||
here's an example |
|||
``` |
|||
kargo [aws|gce] --nodes 2 --etcd 3 --cluster-name test-smana |
|||
``` |
|||
|
|||
#### Deploy kubernetes |
|||
|
|||
With kargo-cli |
|||
``` |
|||
kargo deploy [--aws|--gce] -u admin |
|||
``` |
|||
|
|||
Or ansible-playbook command |
|||
``` |
|||
ansible-playbook -u smana -e ansible_ssh_user=admin -e cloud_provider=[aws|gce] -b --become-user=root -i inventory/single.cfg cluster.yml |
|||
``` |
|||
@ -0,0 +1,24 @@ |
|||
CoreOS bootstrap |
|||
=============== |
|||
|
|||
Example with **kargo-cli**: |
|||
|
|||
``` |
|||
kargo deploy --gce --coreos |
|||
``` |
|||
|
|||
Or with Ansible: |
|||
|
|||
Before running the cluster playbook you must satisfy the following requirements: |
|||
|
|||
* On each CoreOS nodes a writable directory **/opt/bin** (~400M disk space) |
|||
|
|||
* Uncomment the variable **ansible\_python\_interpreter** in the file `inventory/group_vars/all.yml` |
|||
|
|||
* run the Python bootstrap playbook |
|||
|
|||
``` |
|||
ansible-playbook -u smana -e ansible_ssh_user=smana -b --become-user=root -i inventory/inventory.cfg coreos-bootstrap.yml |
|||
``` |
|||
|
|||
Then you can proceed to [cluster deployment](#run-deployment) |
|||
@ -0,0 +1,75 @@ |
|||
 |
|||
|
|||
##Deploy a production ready kubernetes cluster |
|||
|
|||
If you have questions, you can [invite yourself](https://slack.kubespray.io/) to **chat** with us on Slack! [](https://kubespray.slack.com) |
|||
|
|||
- Can be deployed on **AWS, GCE, OpenStack or Baremetal** |
|||
- **High available** cluster |
|||
- **Composable** (Choice of the network plugin for instance) |
|||
- Support most popular **Linux distributions** |
|||
- **Continuous integration tests** |
|||
|
|||
|
|||
To deploy the cluster you can use : |
|||
|
|||
[**kargo-cli**](https://github.com/kubespray/kargo-cli) <br> |
|||
**Ansible** usual commands <br> |
|||
**vagrant** by simply running `vagrant up` (for tests purposes) <br> |
|||
|
|||
|
|||
* [Requirements](#requirements) |
|||
* [Getting started](docs/getting-started.md) |
|||
* [Vagrant install](docs/vagrant.md) |
|||
* [CoreOS bootstrap](docs/coreos.md) |
|||
* [Ansible variables](docs/ansible.md) |
|||
* [Cloud providers](docs/cloud.md) |
|||
* [Openstack](docs/openstack.md) |
|||
* [Network plugins](#network-plugins) |
|||
|
|||
Supported Linux distributions |
|||
=============== |
|||
|
|||
* **CoreOS** |
|||
* **Debian** Wheezy, Jessie |
|||
* **Ubuntu** 14.10, 15.04, 15.10, 16.04 |
|||
* **Fedora** 23 |
|||
* **CentOS/RHEL** 7 |
|||
|
|||
Versions |
|||
-------------- |
|||
|
|||
[kubernetes](https://github.com/kubernetes/kubernetes/releases) v1.3.0 <br> |
|||
[etcd](https://github.com/coreos/etcd/releases) v3.0.1 <br> |
|||
[calicoctl](https://github.com/projectcalico/calico-docker/releases) v0.20.0 <br> |
|||
[flanneld](https://github.com/coreos/flannel/releases) v0.5.5 <br> |
|||
[weave](http://weave.works/) v1.5.0 <br> |
|||
[docker](https://www.docker.com/) v1.10.3 <br> |
|||
|
|||
|
|||
Requirements |
|||
-------------- |
|||
|
|||
* The target servers must have **access to the Internet** in order to pull docker images. |
|||
* The **firewalls are not managed**, you'll need to implement your own rules the way you used to. |
|||
in order to avoid any issue during deployment you should disable your firewall |
|||
* **Copy your ssh keys** to all the servers part of your inventory. |
|||
* **Ansible v2.x and python-netaddr** |
|||
|
|||
|
|||
## Network plugins |
|||
You can choose between 3 network plugins. (default: `flannel` with vxlan backend) |
|||
|
|||
* [**flannel**](docs/flannel.md): gre/vxlan (layer 2) networking. |
|||
|
|||
* [**calico**](docs/calico.md): bgp (layer 3) networking. |
|||
|
|||
* **weave**: Weave is a lightweight container overlay network that doesn't require an external K/V database cluster. <br> |
|||
(Please refer to `weave` [troubleshooting documentation](http://docs.weave.works/weave/latest_release/troubleshooting.html)) |
|||
|
|||
The choice is defined with the variable `kube_network_plugin` |
|||
|
|||
|
|||
[](https://travis-ci.org/kubespray/kargo) </br> |
|||
CI tests sponsored by Google (GCE), and [teuto.net](https://teuto.net/) for OpenStack. |
|||
|
|||
@ -0,0 +1,51 @@ |
|||
Flannel |
|||
============== |
|||
|
|||
* Flannel configuration file should have been created there |
|||
|
|||
``` |
|||
cat /run/flannel/subnet.env |
|||
FLANNEL_NETWORK=10.233.0.0/18 |
|||
FLANNEL_SUBNET=10.233.16.1/24 |
|||
FLANNEL_MTU=1450 |
|||
FLANNEL_IPMASQ=false |
|||
``` |
|||
|
|||
* Check if the network interface has been created |
|||
|
|||
``` |
|||
ip a show dev flannel.1 |
|||
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default |
|||
link/ether e2:f3:a7:0f:bf:cb brd ff:ff:ff:ff:ff:ff |
|||
inet 10.233.16.0/18 scope global flannel.1 |
|||
valid_lft forever preferred_lft forever |
|||
inet6 fe80::e0f3:a7ff:fe0f:bfcb/64 scope link |
|||
valid_lft forever preferred_lft forever |
|||
``` |
|||
|
|||
* Docker must be configured with a bridge ip in the flannel subnet. |
|||
|
|||
``` |
|||
ps aux | grep docker |
|||
root 20196 1.7 2.7 1260616 56840 ? Ssl 10:18 0:07 /usr/bin/docker daemon --bip=10.233.16.1/24 --mtu=1450 |
|||
``` |
|||
|
|||
* Try to run a container and check its ip address |
|||
|
|||
``` |
|||
kubectl run test --image=busybox --command -- tail -f /dev/null |
|||
replicationcontroller "test" created |
|||
|
|||
kubectl describe po test-34ozs | grep ^IP |
|||
IP: 10.233.16.2 |
|||
``` |
|||
|
|||
``` |
|||
kubectl exec test-34ozs -- ip a show dev eth0 |
|||
8: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue |
|||
link/ether 02:42:0a:e9:2b:03 brd ff:ff:ff:ff:ff:ff |
|||
inet 10.233.16.2/24 scope global eth0 |
|||
valid_lft forever preferred_lft forever |
|||
inet6 fe80::42:aff:fee9:2b03/64 scope link tentative flags 08 |
|||
valid_lft forever preferred_lft forever |
|||
``` |
|||
@ -0,0 +1,19 @@ |
|||
Getting started |
|||
=============== |
|||
|
|||
The easiest way to run the deployement is to use the **kargo-cli** tool. |
|||
A complete documentation can be found in its [github repository](https://github.com/kubespray/kargo-cli). |
|||
|
|||
Here is a simple example on AWS: |
|||
|
|||
* Create instances and generate the inventory |
|||
|
|||
``` |
|||
kargo aws --instances 3 |
|||
``` |
|||
|
|||
* Run the deployment |
|||
|
|||
``` |
|||
kargo deploy --aws -u centos -n calico |
|||
``` |
|||
@ -0,0 +1,48 @@ |
|||
OpenStack |
|||
=============== |
|||
|
|||
To deploy kubespray on [OpenStack](https://www.openstack.org/) uncomment the `cloud_provider` option in `group_vars/all.yml` and set it to `'openstack'`. |
|||
|
|||
After that make sure to source in your OpenStack credentials like you would do when using `nova-client` by using `source path/to/your/openstack-rc`. |
|||
|
|||
The next step is to make sure the hostnames in your `inventory` file are identical to your instance names in OpenStack. |
|||
Otherwise [cinder](https://wiki.openstack.org/wiki/Cinder) won't work as expected. |
|||
|
|||
Unless you are using calico you can now run the playbook. |
|||
|
|||
**Additional step needed when using calico:** |
|||
|
|||
Calico does not encapsulate all packages with the hosts ip addresses. Instead the packages will be routed with the PODs ip addresses directly. |
|||
OpenStack will filter and drop all packages from ips it does not know to prevent spoofing. |
|||
|
|||
In order to make calico work on OpenStack you will need to tell OpenStack to allow calicos packages by allowing the network it uses. |
|||
|
|||
First you will need the ids of your OpenStack instances that will run kubernetes: |
|||
|
|||
nova list --tenant Your-Tenant |
|||
+--------------------------------------+--------+----------------------------------+--------+-------------+ |
|||
| ID | Name | Tenant ID | Status | Power State | |
|||
+--------------------------------------+--------+----------------------------------+--------+-------------+ |
|||
| e1f48aad-df96-4bce-bf61-62ae12bf3f95 | k8s-1 | fba478440cb2444a9e5cf03717eb5d6f | ACTIVE | Running | |
|||
| 725cd548-6ea3-426b-baaa-e7306d3c8052 | k8s-2 | fba478440cb2444a9e5cf03717eb5d6f | ACTIVE | Running | |
|||
|
|||
Then you can use the instance ids to find the connected [neutron](https://wiki.openstack.org/wiki/Neutron) ports: |
|||
|
|||
neutron port-list -c id -c device_id |
|||
+--------------------------------------+--------------------------------------+ |
|||
| id | device_id | |
|||
+--------------------------------------+--------------------------------------+ |
|||
| 5662a4e0-e646-47f0-bf88-d80fbd2d99ef | e1f48aad-df96-4bce-bf61-62ae12bf3f95 | |
|||
| e5ae2045-a1e1-4e99-9aac-4353889449a7 | 725cd548-6ea3-426b-baaa-e7306d3c8052 | |
|||
|
|||
Given the port ids on the left, you can set the `allowed_address_pairs` in neutron: |
|||
|
|||
# allow kube_service_addresses network |
|||
neutron port-update 5662a4e0-e646-47f0-bf88-d80fbd2d99ef --allowed_address_pairs list=true type=dict ip_address=10.233.0.0/18 |
|||
neutron port-update e5ae2045-a1e1-4e99-9aac-4353889449a7 --allowed_address_pairs list=true type=dict ip_address=10.233.0.0/18 |
|||
|
|||
# allow kube_pods_subnet network |
|||
neutron port-update 5662a4e0-e646-47f0-bf88-d80fbd2d99ef --allowed_address_pairs list=true type=dict ip_address=10.233.64.0/18 |
|||
neutron port-update e5ae2045-a1e1-4e99-9aac-4353889449a7 --allowed_address_pairs list=true type=dict ip_address=10.233.64.0/18 |
|||
|
|||
Now you can finally run the playbook. |
|||
@ -0,0 +1,41 @@ |
|||
Vagrant Install |
|||
================= |
|||
|
|||
Assuming you have Vagrant (1.8+) installed with virtualbox (it may work |
|||
with vmware, but is untested) you should be able to launch a 3 node |
|||
Kubernetes cluster by simply running `$ vagrant up`.<br /> |
|||
|
|||
This will spin up 3 VMs and install kubernetes on them. Once they are |
|||
completed you can connect to any of them by running <br /> |
|||
`$ vagrant ssh k8s-0[1..3]`. |
|||
|
|||
``` |
|||
$ vagrant up |
|||
Bringing machine 'k8s-01' up with 'virtualbox' provider... |
|||
Bringing machine 'k8s-02' up with 'virtualbox' provider... |
|||
Bringing machine 'k8s-03' up with 'virtualbox' provider... |
|||
==> k8s-01: Box 'bento/ubuntu-14.04' could not be found. Attempting to find and install... |
|||
... |
|||
... |
|||
k8s-03: Running ansible-playbook... |
|||
|
|||
PLAY [k8s-cluster] ************************************************************* |
|||
|
|||
TASK [setup] ******************************************************************* |
|||
ok: [k8s-03] |
|||
ok: [k8s-01] |
|||
ok: [k8s-02] |
|||
... |
|||
... |
|||
PLAY RECAP ********************************************************************* |
|||
k8s-01 : ok=157 changed=66 unreachable=0 failed=0 |
|||
k8s-02 : ok=137 changed=59 unreachable=0 failed=0 |
|||
k8s-03 : ok=86 changed=51 unreachable=0 failed=0 |
|||
|
|||
$ vagrant ssh k8s-01 |
|||
vagrant@k8s-01:~$ kubectl get nodes |
|||
NAME STATUS AGE |
|||
k8s-01 Ready 45s |
|||
k8s-02 Ready 45s |
|||
k8s-03 Ready 45s |
|||
``` |
|||
Write
Preview
Loading…
Cancel
Save