Added file and container image caching (#4828)

* File and container image downloads are now cached localy, so that repeated vagrant up/down runs do not trigger downloading of those files. This is especially useful on laptops with kubernetes runnig locally on vm's. The total size of the cache, after an ansible run, is currently around 800MB, so bandwidth (=time) savings can be quite significant.

* When download_run_once is false, the default is still not to cache, but setting download_force_cache will still enable caching.

* The local cache location can be set with download_cache_dir and defaults to /tmp/kubernetes_cache

* A local docker instance is no longer required to cache docker images; Images are cached to file. A local docker instance is still required, though, if you wish to download images on localhost.

* Fixed a FIXME, wher the argument was that delegate_to doesn't play nice with omit. That is a correct observation and the fix is to use default(inventory_host) instead of default(omit). See ansible/ansible#26009

* Removed "Register docker images info" task from download_container and set_docker_image_facts because it was faulty and unused.

* Removed redundant when:download.{container,enabled,run_once} conditions from {sync,download}_container.yml

* All features of commit d6fd0d2aca by Timoses <timosesu@gmail.com>, merged May 1st 2019, are included in this patch. Not all code was included verbatim, but each feature of that commit was checked to be working in this patch. One notable change: The actual downloading of the kubeadm images was moved to {download,sync)_container, to enable caching.

Note 1: I considered splitting this patch, but most changes that are not directly related to caching, are a pleasant by-product of implementing the caching code, so splitting would be impractical.

Note 2: I have my doubts about the usefulness of the upload, download and upgrade tags in the download role. Must they remain or can they be removed? If anybody knows, then please speak up.

Vagrantfile

@@ -21,7 +21,7 @@ SUPPORTED_OS = {
   "ubuntu1604"          => {box: "generic/ubuntu1604", user: "vagrant"},
   "ubuntu1804"          => {box: "generic/ubuntu1804", user: "vagrant"},
   "centos"              => {box: "centos/7",           user: "vagrant"},
-  "centos-bento"        => {box: "bento/centos-7.5",   user: "vagrant"},
+  "centos-bento"        => {box: "bento/centos-7.6",   user: "vagrant"},
   "fedora"              => {box: "fedora/28-cloud-base",                user: "vagrant"},
   "opensuse"            => {box: "opensuse/openSUSE-15.0-x86_64",       user: "vagrant"},
   "opensuse-tumbleweed" => {box: "opensuse/openSUSE-Tumbleweed-x86_64", user: "vagrant"},
@@ -180,9 +180,17 @@ Vagrant.configure("2") do |config|
         "flannel_interface": "eth1",
         "kube_network_plugin": $network_plugin,
         "kube_network_plugin_multus": $multi_networking,
-        "docker_keepcache": "1",
-        "download_run_once": "False",
+        "download_run_once": "True",
         "download_localhost": "False",
+        "download_cache_dir": ENV['HOME'] + "/kubespray_cache",
+        # Make kubespray cache even when download_run_once is false
+        "download_force_cache": "True",
+        # Keeping the cache on the nodes can improve provisioning speed while debugging kubespray
+        "download_keep_remote_cache": "False",
+        "docker_keepcache": "1",
+        # These two settings will put kubectl and admin.config in $inventory/artifacts
+        "kubeconfig_localhost": "True",
+        "kubectl_localhost": "True",
         "local_path_provisioner_enabled": "#{$local_path_provisioner_enabled}",
         "local_path_provisioner_claim_root": "#{$local_path_provisioner_claim_root}",
         "ansible_ssh_user": SUPPORTED_OS[$os][:user]

docs/downloads.md

@@ -3,23 +3,22 @@ Downloading binaries and containers
 
 Kubespray supports several download/upload modes. The default is:
 
-* Each node downloads binaries and container images on its own, which is
-  ``download_run_once: False``.
+* Each node downloads binaries and container images on its own, which is ``download_run_once: False``.
 * For K8s apps, pull policy is ``k8s_image_pull_policy: IfNotPresent``.
-* For system managed containers, like kubelet or etcd, pull policy is
-  ``download_always_pull: False``, which is pull if only the wanted repo and
-  tag/sha256 digest differs from that the host has.
+* For system managed containers, like kubelet or etcd, pull policy is ``download_always_pull: False``, which is pull if only the wanted repo and tag/sha256 digest differs from that the host has.
 
 There is also a "pull once, push many" mode as well:
 
-* Override the ``download_run_once: True`` to download container images and binaries only once
-  then push to cluster nodes in batches. The default delegate node
-  for pushing is the first `kube-master`.
-* If your ansible runner node (aka the admin node) have password-less sudo and
-  docker enabled, you may want to define the ``download_localhost: True``, which
-  makes that node a delegate for pushing while running the deployment with
-  ansible. This may be the case if cluster nodes cannot access each other via ssh
-  or you want to use local docker images and binaries as a cache for multiple clusters.
+* Setting ``download_run_once: True`` will make kubespray download container images and binaries only once and then push them to the cluster nodes. The default download delegate node is the first `kube-master`.
+* Set ``download_localhost: True`` to make localhost the download delegate. This can be useful if cluster nodes cannot access external addresses. To use this requires that docker is installed and running on the ansible master and that the current user is either in the docker group or can do passwordless sudo, to be able to access docker.
+
+NOTE: When download_once is true and download_localhost is false, all downloads will be done on the delegate node, including downloads for container images that are not required on that node. As a consequence, the storage required on that node will probably be more than if download_run_once was false, because all images will be loaded into the docker instance on that node, instead of just the images required for that node.
+
+On caching:
+
+* When download_once is true, all downloaded files will be cached locally in $download_cache_dir, which defaults to /tmp/kubespray_cache. On subsequent provisioning runs, this local cache will be used to provision the nodes, minimizing bandwidth usage and improving provisining time. Expect about 800MB of disk space to be used on the ansible node for the cache. Disk space required for the image cache on the kubernetes nodes is a much as is needed for the largest image, which is currently slightly less than 150MB.
+* By default, if download_once is false, kubespray will not retreive the downloaded images and files from the remote node to the local cache, or use that cache to pre-provision those nodes. To force the use of the cache, set download_force_cache to true.
+* By default, cached images that are used to pre-provision the remote nodes will be deleted from the remote nodes after use, to save disk space. Setting download_keep_remote_cache will prevent the files from being deleted. This can be useful while developping kubespray, as it can decrease provisioning times. As a consequence, the required storage for images on the remote nodes will increase from 150MB to about 550MB, which is currently the combined size of all required container images.
 
 Container images and binary files are described by the vars like ``foo_version``,
 ``foo_download_url``, ``foo_checksum`` for binaries and ``foo_image_repo``,
@@ -36,8 +35,7 @@ dnsmasq_digest_checksum: 7c883354f6ea9876d176fe1d30132515478b2859d6fc0cbf9223ffd
 dnsmasq_image_repo: andyshinn/dnsmasq
 dnsmasq_image_tag: '2.72'
 ```
-The full list of available vars may be found in the download's ansible role defaults.
-Those also allow to specify custom urls and local repositories for binaries and container
+The full list of available vars may be found in the download's ansible role defaults. Those also allow to specify custom urls and local repositories for binaries and container
 images as well. See also the DNS stack docs for the related intranet configuration,
 so the hosts can resolve those urls and repos.
 
@@ -46,7 +44,7 @@ so the hosts can resolve those urls and repos.
 In case your servers don't have access to internet (for example when deploying on premises with security constraints), you'll have, first, to setup the appropriate proxies/caches/mirrors and/or internal repositories and registries and, then, adapt the following variables to fit your environment before deploying:
 
 * At least `foo_image_repo` and `foo_download_url` as described before (i.e. in case of use of proxies to registries and binaries repositories, checksums and versions do not necessarily need to be changed).
-  NB: Regarding `foo_image_repo`, when using insecure registries/proxies, you will certainly have to append them to the `docker_insecure_registries` variable in group_vars/all/docker.yml
+  NOTE: Regarding `foo_image_repo`, when using insecure registries/proxies, you will certainly have to append them to the `docker_insecure_registries` variable in group_vars/all/docker.yml
 * `pyrepo_index` (and optionally `pyrepo_cert`)
 * Depending on the `container_manager`
   * When `container_manager=docker`, `docker_foo_repo_base_url`, `docker_foo_repo_gpgkey`, `dockerproject_bar_repo_base_url` and `dockerproject_bar_repo_gpgkey` (where `foo` is the distribution and `bar` is system package manager)

roles/download/defaults/main.yml

@@ -1,5 +1,15 @@
 ---
 local_release_dir: /tmp/releases
+download_cache_dir: /tmp/kubespray_cache
+
+# do not delete remote cache files after using them
+# NOTE: Setting this parameter to TRUE is only really useful when developing kubespray
+download_keep_remote_cache: false
+
+# Only useful when download_run_once is false: Localy cached files and images are
+# uploaded to kubernetes nodes. Also, images downloaded on those nodes are copied
+# back to the ansible runner's cache, if they are not yet preset.
+download_force_cache: false
 
 # Used to only evaluate vars from download role
 skip_downloads: false

roles/download/tasks/check_pull_required.yml

@@ -0,0 +1,33 @@
+---
+# NOTE: The ampersand hell in this block is needed because docker-inspect uses go templates,
+# which uses double ampersands as delimeters, just like Jinja does. If you want to understand
+# the template, just replace all instances  of {{ `{{` }} with {{ and {{ '}}' }} with }}.
+# It will output something like the following:
+# nginx:1.15,gcr.io/google-containers/kube-proxy:v1.14.1,gcr.io/google-containers/kube-proxy@sha256:44af2833c6cbd9a7fc2e9d2f5244a39dfd2e31ad91bf9d4b7d810678db738ee9,gcr.io/google-containers/kube-apiserver:v1.14.1,etc...
+- name: check_pull_required |  Generate a list of information about the images on a node
+  shell: >-
+    {{ docker_bin_dir }}/docker images -q | xargs -r {{ docker_bin_dir }}/docker inspect -f "{{ '{{' }} if .RepoTags {{ '}}' }}{{ '{{' }} (index .RepoTags) {{ '}}' }}{{ '{{' }} end {{ '}}' }}{{ '{{' }} if .RepoDigests {{ '}}' }},{{ '{{' }} (index .RepoDigests) {{ '}}' }}{{ '{{' }} end {{ '}}' }}" | sed -e 's/^ *\[//g' -e 's/\] *$//g' -e 's/ /\n/g' | tr '\n' ','
+  delegate_to: "{{ download_delegate if download_run_once or inventory_hostname }}"
+  no_log: true
+  register: docker_images
+  failed_when: false
+  changed_when: false
+  check_mode: no
+  become: "{{ not download_localhost }}"
+  when: not download_always_pull
+
+- name: check_pull_required | Set pull_required if the desired image is not yet loaded
+  set_fact:
+    pull_required: >-
+      {%- if image_reponame in docker_images.stdout.split(',') %}false{%- else -%}true{%- endif -%}
+  when: not download_always_pull
+
+- name: check_pull_required | Check that the local digest sha256 corresponds to the given image tag
+  assert:
+    that: "{{ download.repo }}:{{ download.tag }} in docker_images.stdout.split(',')"
+  when:
+    - not download_always_pull
+    - not pull_required
+    - pull_by_digest
+  tags:
+    - asserts

roles/download/tasks/download_container.yml

@@ -1,40 +1,129 @@
 ---
-- name: container_download | Make download decision if pull is required by tag or sha256
-  include_tasks: set_docker_image_facts.yml
-  when:
-    - download.enabled
-    - download.container
-  tags:
+- block:
+  - name: download_container | Set a few facts
+    import_tasks: set_container_facts.yml
+    run_once: "{{ download_run_once }}"
+    tags:
+    - facts
+
+  - name: download_container | Determine if image is in cache
+    stat:
+      path: "{{ image_path_cached }}"
+    delegate_to: localhost
+    delegate_facts: no
+    register: cache_image
+    changed_when: false
+    become: false
+    when:
+    - download_force_cache
+
+  - name: download_container | Set fact indicating if image is in cache
+    set_fact:
+      image_is_cached: "{{ cache_image.stat.exists | default(false) }}"
+    tags:
     - facts
+    when:
+    - download_force_cache
+
+  - name: download_container | Upload image to node if it is cached
+    synchronize:
+      src: "{{ image_path_cached }}"
+      dest: "{{ image_path_final }}"
+      use_ssh_args: "{{ has_bastion | default(false) }}"
+      mode: push
+    delegate_facts: no
+    register: upload_image
+    failed_when: not upload_image
+    run_once: "{{ download_run_once }}"
+    until: upload_image is succeeded
+    retries: 4
+    delay: "{{ retry_stagger | random + 3 }}"
+    when:
+    - download_force_cache
+    - image_is_cached
+    - not download_localhost
+    - ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"]
+
+  - name: download_container | Load image into docker
+    shell: "{{ docker_bin_dir }}/docker load < {{ image_path_cached if download_localhost else image_path_final }}"
+    delegate_to: "{{ download_delegate if download_run_once or inventory_hostname }}"
+    run_once: "{{ download_run_once }}"
+    register: container_load_status
+    failed_when: container_load_status | failed
+    become: "{{ user_can_become_root | default(false) or not (download_run_once and download_localhost) }}"
+    when:
+    - download_force_cache
+    - image_is_cached
+    - ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"]
+
+  - name: download_container | Prepare container download
+    import_tasks: check_pull_required.yml
+    run_once: "{{ download_run_once }}"
+    when:
+    - not download_always_pull
+
+  - debug:
+      msg: "XXX Pull required is: {{ pull_required }}"
 
-# FIXME(mattymo): In Ansible 2.4 omitting download delegate is broken. Move back
-#                 to one task in the future.
-- name: container_download | Download containers if pull is required or told to always pull (delegate)
-  command: "{{ docker_bin_dir }}/docker pull {{ pull_args }}"
-  register: pull_task_result
-  until: pull_task_result is succeeded
-  retries: 4
-  delay: "{{ retry_stagger | random + 3 }}"
-  changed_when: not 'up to date' in pull_task_result.stdout
-  when:
-    - download_run_once
-    - download.enabled
-    - download.container
-    - any_pull_required | default(download_always_pull)
-  delegate_to: "{{ download_delegate }}"
-  delegate_facts: yes
-  run_once: yes
-
-- name: container_download | Download containers if pull is required or told to always pull (all nodes)
-  command: "{{ docker_bin_dir }}/docker pull {{ pull_args }}"
-  register: pull_task_result
-  until: pull_task_result is succeeded
-  retries: 4
-  delay: "{{ retry_stagger | random + 3 }}"
-  changed_when: not 'up to date' in pull_task_result.stdout
-  when:
-    - not download_run_once
-    - download.enabled
-    - download.container
-    - pull_required|default(download_always_pull)
-    - group_names | intersect(download.groups) | length
+  # NOTE: Pre-loading docker images will not prevent 'docker pull' from re-downloading the layers in that image
+  # if a pull is forced. This is a known issue with docker. See https://github.com/moby/moby/issues/23684
+  - name: download_container | Download image if required
+    command: "{{ docker_bin_dir }}/docker pull {{ image_reponame }}"
+    delegate_to: "{{ download_delegate if download_run_once or inventory_hostname }}"
+    delegate_facts: yes
+    run_once: "{{ download_run_once }}"
+    register: pull_task_result
+    until: pull_task_result is succeeded
+    delay: "{{ retry_stagger | random + 3 }}"
+    retries: 4
+    become: "{{ user_can_become_root | default(false) or not download_localhost }}"
+    when:
+    - pull_required | default(download_always_pull)
+
+  # NOTE: image_changed is only valid if a pull is was needed or forced.
+  - name: download_container | Check if image changed
+    set_fact:
+      image_changed: "{{ true if pull_task_result.stdout is defined and not 'up to date' in pull_task_result.stdout else false }}"
+    run_once: true
+    when:
+    - download_force_cache
+    tags:
+    - facts
+
+  - name: download_container | Save and compress image
+    shell: "{{ docker_bin_dir }}/docker save {{ image_reponame }} | gzip -{{ download_compress }} > {{ image_path_cached if download_localhost else image_path_final }}"
+    delegate_to: "{{ download_delegate if download_run_once or inventory_hostname }}"
+    delegate_facts: no
+    register: container_save_status
+    failed_when: container_save_status.stderr
+    run_once: true
+    become: "{{ user_can_become_root | default(false) or not download_localhost }}"
+    when:
+    - download_force_cache
+    - not image_is_cached or (image_changed | default(true))
+    - ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"]
+
+  - name: download_container | Copy image to ansible host cache
+    synchronize:
+      src: "{{ image_path_final }}"
+      dest: "{{ image_path_cached }}"
+      use_ssh_args: "{{ has_bastion | default(false) }}"
+      mode: pull
+    delegate_facts: no
+    run_once: true
+    when:
+    - download_force_cache
+    - not download_localhost
+    - not image_is_cached or (image_changed | default(true))
+    - ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"]
+
+  - name: download_container | Remove container image from cache
+    file:
+      state: absent
+      path: "{{ image_path_final }}"
+    when:
+    - not download_keep_remote_cache
+    - ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"]
+
+  tags:
+  - download

roles/download/tasks/download_file.yml

@@ -1,76 +1,123 @@
 ---
-- name: file_download | Downloading...
-  debug:
-    msg:
-      - "URL: {{ download.url }}"
-      - "Dest: {{ download.dest }}"
+- block:
+  - name: download_file | Starting dowload of file
+    debug:
+      msg: "{{ download.url }}"
+    run_once: "{{ download_run_once }}"
 
-- name: file_download | Create dest directory
-  file:
-    path: "{{ download.dest | dirname }}"
-    state: directory
-    recurse: yes
-  when:
-    - download.enabled
-    - download.file
-    - group_names | intersect(download.groups) | length
+  - name: download_file | Set pathname of cached file
+    set_fact:
+      file_path_cached: "{{ download_cache_dir }}/{{ download.dest | regex_replace('^\\/', '') }}"
+    tags:
+    - facts
 
-# As in 'download_container.yml':
-#   In Ansible 2.4 omitting download delegate is broken. Move back
-#   to one task in the future.
-- name: file_download | Download item (delegate)
-  get_url:
-    url: "{{ download.url }}"
-    dest: "{{ download.dest }}"
-    sha256sum: "{{ download.sha256|default(omit) }}"
-    owner: "{{ download.owner|default(omit) }}"
-    mode: "{{ download.mode|default(omit) }}"
-    validate_certs: "{{ download_validate_certs }}"
-    url_username: "{{ download.username|default(omit) }}"
-    url_password: "{{ download.password|default(omit) }}"
-    force_basic_auth: "{{ download.force_basic_auth|default(omit) }}"
-  register: get_url_result
-  until: "'OK' in get_url_result.msg or 'file already exists' in get_url_result.msg"
-  retries: 4
-  delay: "{{ retry_stagger | default(5) }}"
-  delegate_to: "{{ download_delegate }}"
-  when:
-    - download_run_once
-    - download.enabled
-    - download.file
-    - group_names | intersect(download.groups) | length
-  run_once: yes
+  - name: download_file | Create dest directory on node
+    file:
+      path: "{{ download.dest | dirname }}"
+      owner: "{{ download.owner | default(omit) }}"
+      mode: 0755
+      state: directory
+      recurse: yes
 
-- name: file_download | Download item (all)
-  get_url:
-    url: "{{ download.url }}"
-    dest: "{{ download.dest }}"
-    sha256sum: "{{ download.sha256|default(omit) }}"
-    owner: "{{ download.owner|default(omit) }}"
-    mode: "{{ download.mode|default(omit) }}"
-    validate_certs: "{{ download_validate_certs }}"
-    url_username: "{{ download.username|default(omit) }}"
-    url_password: "{{ download.password|default(omit) }}"
-    force_basic_auth: "{{ download.force_basic_auth|default(omit) }}"
-  register: get_url_result
-  until: "'OK' in get_url_result.msg or 'file already exists' in get_url_result.msg"
-  retries: 4
-  delay: "{{ retry_stagger | default(5) }}"
-  when:
-    - not download_run_once
-    - download.enabled
-    - download.file
-    - group_names | intersect(download.groups) | length
+  - name: download_file | Create local cache directory
+    file:
+      path: "{{ file_path_cached | dirname }}"
+      state: directory
+      recurse: yes
+    delegate_to: localhost
+    delegate_facts: false
+    run_once: true
+    become: false
+    tags:
+    - localhost
 
-- name: file_download | Extract archives
-  unarchive:
-    src: "{{ download.dest }}"
-    dest: "{{ download.dest |dirname }}"
-    owner: "{{ download.owner|default(omit) }}"
-    mode: "{{ download.mode|default(omit) }}"
-    copy: no
-  when:
-    - download.enabled
-    - download.file
-    - download.unarchive|default(False)
-    - group_names | intersect(download.groups) | length
+  - name: download_file | Check if file is available in cache
+    stat:
+      path: "{{ file_path_cached }}"
+    register: cache_file
+    run_once: true
+    changed_when: false
+    delegate_to: localhost
+    delegate_facts: no
+    become: false
+    when:
+    - download_force_cache
+    tags:
+    - facts
+
+  - name: download_file | Set file_is_cached fact based on previous task
+    set_fact:
+      file_is_cached: "{{ cache_file.stat.exists | default(false) }}"
+    when:
+    - download_force_cache
+    tags:
+    - facts
+
+  - name: download_file | Copy file from cache to nodes, if it is available
+    synchronize:
+      src: "{{ file_path_cached }}"
+      dest: "{{ download.dest }}"
+      use_ssh_args: "{{ has_bastion | default(false) }}"
+      mode: push
+    run_once: "{{ download_run_once }}"
+    register: get_task
+    until: get_task is succeeded
+    delay: "{{ retry_stagger | random + 3 }}"
+    retries: 4
+    when:
+    - download_force_cache
+    - file_is_cached
+    - ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"]
+
+  - name: download_file | Set mode and owner
+    file:
+      path: "{{ download.dest }}"
+      mode: "{{ download.mode | default(omit) }}"
+      owner: "{{ download.owner | default(omit) }}"
+    run_once: "{{ download_run_once }}"
+    when:
+    - download_force_cache
+    - file_is_cached
+    - ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"]
+
+  # This must always be called, to check if the checksum matches. On no-match the file is re-downloaded.
+  - name: download_file | Download item
+    get_url:
+      url: "{{ download.url }}"
+      dest: "{{ file_path_cached if download_localhost else download.dest }}"
+      owner: "{{ omit if download_localhost else (download.owner | default(omit)) }}"
+      mode: "{{ omit if download_localhost else (download.mode | default(omit)) }}"
+      checksum: "{{ 'sha256:' + download.sha256 if download.sha256 or omit }}"
+      validate_certs: "{{ download_validate_certs }}"
+      url_username: "{{ download.username | default(omit) }}"
+      url_password: "{{ download.password | default(omit) }}"
+      force_basic_auth: "{{ download.force_basic_auth | default(omit) }}"
+    delegate_to: "{{ download_delegate if download_run_once else inventory_hostname }}"
+    run_once: "{{ download_run_once }}"
+    register: get_url_result
+    become: "{{ not download_localhost }}"
+    until: "'OK' in get_url_result.msg or 'file already exists' in get_url_result.msg"
+    retries: 4
+    delay: "{{ retry_stagger | default(5) }}"
+
+  - name: "download_file | Extract file archives"
+    include_tasks: "extract_file.yml"
+    when:
+    - not download_localhost
+
+  - name: download_file | Copy file back to ansible host file cache
+    synchronize:
+      src: "{{ download.dest }}"
+      dest: "{{ file_path_cached }}"
+      use_ssh_args: "{{ has_bastion | default(false) }}"
+      mode: pull
+    run_once: true
+    when:
+    - download_force_cache
+    - not file_is_cached or get_url_result.changed
+    - download_delegate == inventory_hostname
+    - not (download_run_once and download_delegate == 'localhost')
+    - ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"]
+
+  tags:
+  - download

roles/download/tasks/download_prep.yml

@@ -1,35 +0,0 @@
----
-- name: Register docker images info
-  shell: >-
-    {{ docker_bin_dir }}/docker images -q | xargs {{ docker_bin_dir }}/docker inspect -f "{{ '{{' }} (index .RepoTags 0) {{ '}}' }},{{ '{{' }} (index .RepoDigests 0) {{ '}}' }}" | tr '\n' ','
-  no_log: true
-  register: docker_images
-  failed_when: false
-  changed_when: false
-  check_mode: no
-  when: download_container
-
-- name: container_download | Create dest directory for saved/loaded container images
-  file:
-    path: "{{ local_release_dir }}/containers"
-    state: directory
-    recurse: yes
-    mode: 0755
-    owner: "{{ ansible_ssh_user|default(ansible_user_id) }}"
-  when: download_container
-
-- name: container_download | create local directory for saved/loaded container images
-  file:
-    path: "{{ local_release_dir }}/containers"
-    state: directory
-    recurse: yes
-  delegate_to: localhost
-  delegate_facts: false
-  become: false
-  run_once: true
-  when:
-    - download_run_once
-    - download_delegate == 'localhost'
-    - download_container
-  tags:
-    - localhost

roles/download/tasks/extract_file.yml

@@ -0,0 +1,10 @@
+---
+- name: extract_file | Unpacking archive
+  unarchive:
+    src: "{{ download.dest }}"
+    dest: "{{ download.dest | dirname }}"
+    owner: "{{ download.owner | default(omit) }}"
+    mode: "{{ download.mode | default(omit) }}"
+    copy: no
+  when:
+    - download.unarchive | default(false)

roles/download/tasks/main.yml

@@ -1,40 +1,56 @@
 ---
-- include_tasks: download_prep.yml
+- name: download | Prepare working directories and variables
+  import_tasks: prep_download.yml
   when:
     - not skip_downloads|default(false)
+  tags:
+    - download
+    - upload
 
-- include_tasks: kubeadm_images.yml
+- name: download | Get kubeadm binary and list of required images
+  import_tasks: prep_kubeadm_images.yml
   when:
     - kube_version is version('v1.11.0', '>=')
     - not skip_downloads|default(false)
     - not skip_kubeadm_images|default(false)
     - inventory_hostname in groups['kube-master']
+  tags:
+    - download
+    - upload
 
-- name: Set kubeadm_images
+- name: download | Create kubeadm_images variable if it is absent
   set_fact:
     kubeadm_images: {}
   when:
     - kubeadm_images is not defined
+  tags:
+    - download
+    - upload
+    - facts
 
-- name: "Download items"
+- name: download | Download files / images
   include_tasks: "{{ include_file }}"
+  with_dict: "{{ downloads | combine(kubeadm_images) }}"
   vars:
     download: "{{ download_defaults | combine(item.value) }}"
     include_file: "download_{% if download.container %}container{% else %}file{% endif %}.yml"
-  with_dict: "{{ downloads | combine(kubeadm_images) }}"
   when:
-    - not skip_downloads|default(false)
+    - not skip_downloads | default(false)
+    - download.enabled
     - item.value.enabled
-    - (not (item.value.container|default(False))) or (item.value.container and download_container)
+    - (not (item.value.container | default(false))) or (item.value.container and download_container)
+    - (download_run_once and inventory_hostname == download_delegate) or (group_names | intersect(download.groups) | length)
 
-- name: "Sync items"
+- name: download | Sync files / images from ansible host to nodes
   include_tasks: "{{ include_file }}"
+  with_dict: "{{ downloads | combine(kubeadm_images) }}"
   vars:
     download: "{{ download_defaults | combine(item.value) }}"
     include_file: "sync_{% if download.container %}container{% else %}file{% endif %}.yml"
-  with_dict: "{{ downloads | combine(kubeadm_images) }}"
   when:
-    - not skip_downloads|default(false)
+    - not skip_downloads | default(false)
+    - download.enabled
     - item.value.enabled
     - download_run_once
     - group_names | intersect(download.groups) | length
+    - not (inventory_hostname == download_delegate)

roles/download/tasks/prep_download.yml

@@ -0,0 +1,78 @@
+---
+- name: prep_download | Set a few facts
+  set_fact:
+    download_force_cache: "{{ true if download_run_once else download_force_cache }}"
+  tags:
+    - facts
+
+- name: prep_download | Create staging directory on remote node
+  file:
+    path: "{{ local_release_dir }}/images"
+    state: directory
+    recurse: yes
+    mode: 0755
+    owner: "{{ ansible_ssh_user | default(ansible_user_id) }}"
+  when:
+    - ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"]
+
+- name: prep_download | Create local cache for files and images
+  file:
+    path: "{{ download_cache_dir }}/images"
+    state: directory
+    recurse: yes
+    mode: 0755
+  delegate_to: localhost
+  delegate_facts: no
+  run_once: true
+  become: false
+  tags:
+    - localhost
+
+- name: prep_download | On localhost, check if passwordless root is possible
+  command: "true"
+  delegate_to: localhost
+  run_once: true
+  register: test_become
+  changed_when: false
+  ignore_errors: true
+  become: true
+  when:
+    - download_localhost
+  tags:
+    - localhost
+    - asserts
+
+- name: prep_download | On localhost, check if user has access to docker without using sudo
+  shell: "{{ docker_bin_dir }}/docker images"
+  delegate_to: localhost
+  run_once: true
+  register: test_docker
+  changed_when: false
+  ignore_errors: true
+  become: false
+  when:
+    - download_localhost
+  tags:
+    - localhost
+    - asserts
+
+- name: prep_download | Parse the outputs of the previous commands
+  set_fact:
+    user_in_docker_group: "{{ not test_docker.failed }}"
+    user_can_become_root: "{{ not test_become.failed }}"
+  when:
+    - download_localhost
+  tags:
+    - localhost
+    - asserts
+
+- name: prep_download | Check that local user is in group or can become root
+  assert:
+    that: "user_in_docker_group or user_can_become_root"
+    msg: >-
+      Error: User is not in docker group and cannot become root. When download_localhost is true, at least one of these two conditions must be met.
+  when:
+    - download_localhost
+  tags:
+    - localhost
+    - asserts

roles/download/tasks/kubeadm_images.yml → roles/download/tasks/prep_kubeadm_images.yml

@@ -1,28 +1,28 @@
 ---
-- name: kubeadm | Download kubeadm
+- name: prep_kubeadm_images | Download kubeadm binary
   include_tasks: "download_file.yml"
   vars:
     download: "{{ download_defaults | combine(downloads.kubeadm) }}"
   when:
-    - not skip_downloads|default(false)
+    - not skip_downloads | default(false)
     - downloads.kubeadm.enabled
 
-- name: kubeadm | Sync kubeadm
+- name: prep_kubeadm_images | Sync kubeadm binary to nodes
   include_tasks: "sync_file.yml"
   vars:
     download: "{{ download_defaults | combine(downloads.kubeadm) }}"
   when:
-    - not skip_downloads|default(false)
+    - not skip_downloads | default(false)
     - downloads.kubeadm.enabled
     - download_run_once
     - group_names | intersect(download.groups) | length
 
-- name: kubeadm | Create kubeadm config
+- name: prep_kubeadm_images | Create kubeadm config
   template:
     src: "kubeadm-images.yaml.j2"
     dest: "{{ kube_config_dir }}/kubeadm-images.yaml"
 
-- name: kubeadm | Copy kubeadm binary from download dir
+- name: prep_kubeadm_images | Copy kubeadm binary from download dir to system path
   synchronize:
     src: "{{ local_release_dir }}/kubeadm-{{ kubeadm_version }}-{{ image_arch }}"
     dest: "{{ bin_dir }}/kubeadm"
@@ -32,26 +32,21 @@
     group: no
   delegate_to: "{{ inventory_hostname }}"
 
-- name: kubeadm | Set kubeadm binary permissions
+- name: prep_kubeadm_images | Set kubeadm binary permissions
   file:
     path: "{{ bin_dir }}/kubeadm"
     mode: "0755"
     state: file
 
-- name: container_download | download images for kubeadm config images
-  command: "{{ bin_dir }}/kubeadm config images pull --config={{ kube_config_dir }}/kubeadm-images.yaml"
-  when: not download_run_once
-
-- name: container_download | fetch list of kubeadm config images
+- name: prep_kubeadm_images | Generate list of required images
   command: "{{ bin_dir }}/kubeadm config images list --config={{ kube_config_dir }}/kubeadm-images.yaml"
-  register: result
+  register: kubeadm_images_raw
   run_once: true
-  when: download_run_once
   changed_when: false
 
-- name: container_download | extract container names from list of kubeadm config images
+- name: prep_kubeadm_images | Parse list of images
   vars:
-    kubeadm_images_list: "{{ result.stdout_lines }}"
+    kubeadm_images_list: "{{ kubeadm_images_raw.stdout_lines }}"
   set_fact:
     kubeadm_image:
       key: "kubeadm_{{ (item | regex_replace('^(?:.*\\/)*','')).split(':')[0] }}"
@@ -60,15 +55,12 @@
         container: true
         repo: "{{ item.split(':')[0] }}"
         tag: "{{ item.split(':')[1] }}"
-        groups:
-          - k8s-cluster
+        groups: k8s-cluster
   loop: "{{ kubeadm_images_list | flatten(levels=1) }}"
+  register: kubeadm_images_cooked
   run_once: true
-  when: download_run_once
-  register: result_images
 
-- name: container_download | set kubeadm_images
+- name: prep_kubeadm_images | Convert list of images to dict for later use
   set_fact:
-    kubeadm_images: "{{ result_images.results | map(attribute='ansible_facts.kubeadm_image') | list | items2dict }}"
+    kubeadm_images: "{{ kubeadm_images_cooked.results | map(attribute='ansible_facts.kubeadm_image') | list | items2dict }}"
   run_once: true
-  when: download_run_once

roles/download/tasks/set_container_facts.yml

@@ -0,0 +1,23 @@
+---
+- name: set_container_facts | Display the name of the image being processed
+  debug:
+    msg: "{{ download.repo }}"
+
+- name: set_container_facts | Set if containers should be pulled by digest
+  set_fact:
+    pull_by_digest: >-
+      {%- if download.sha256 is defined and download.sha256 -%}true{%- else -%}false{%- endif -%}
+
+- name: set_container_facts | Define by what name to pull the image
+  set_fact:
+    image_reponame: >-
+      {%- if pull_by_digest %}{{ download.repo }}@sha256:{{ download.sha256 }}{%- else -%}{{ download.repo }}:{{ download.tag }}{%- endif -%}
+
+- name: set_container_facts | Define file name of image
+  set_fact:
+    image_filename: "{{ image_reponame | regex_replace('/|\0|:', '_') }}.tar"
+
+- name: set_container_facts | Define path of image
+  set_fact:
+    image_path_cached: "{{ download_cache_dir }}/images/{{ image_filename }}"
+    image_path_final: "{{ local_release_dir }}/images/{{ image_filename }}"

roles/download/tasks/set_docker_image_facts.yml

@@ -1,50 +0,0 @@
----
-- name: Set if containers should be pulled by digest
-  set_fact:
-    pull_by_digest: >-
-      {%- if download.sha256 is defined and download.sha256 -%}true{%- else -%}false{%- endif -%}
-
-- name: Set pull_args
-  set_fact:
-    pull_args: >-
-      {%- if pull_by_digest %}{{ download.repo }}@sha256:{{ download.sha256 }}{%- else -%}{{ download.repo }}:{{ download.tag }}{%- endif -%}
-
-- name: Register docker images info
-  shell: >-
-    {{ docker_bin_dir }}/docker images -q | xargs -r {{ docker_bin_dir }}/docker inspect -f "{{ '{{' }} if .RepoTags {{ '}}' }}{{ '{{' }} (index .RepoTags) {{ '}}' }}{{ '{{' }} end {{ '}}' }}{{ '{{' }} if .RepoDigests {{ '}}' }},{{ '{{' }} (index .RepoDigests) {{ '}}' }}{{ '{{' }} end {{ '}}' }}" | sed -e 's/^ *\[//g' -e 's/\] *$//g' -e 's/ /\n/g' | tr '\n' ','
-  no_log: true
-  register: docker_images
-  failed_when: false
-  changed_when: false
-  check_mode: no
-  when:
-    - not download_always_pull
-    - group_names | intersect(download.groups) | length
-
-- name: Set if pull is required per container
-  set_fact:
-    pull_required: >-
-      {%- if pull_args in docker_images.stdout.split(',') %}false{%- else -%}true{%- endif -%}
-  when:
-    - not download_always_pull
-    - group_names | intersect(download.groups) | length
-
-- name: Does any host require container pull?
-  vars:
-    hosts_pull_required: "{{ hostvars.values() | map(attribute='pull_required') | select('defined') | list }}"
-  set_fact:
-    any_pull_required: "{{ True in hosts_pull_required }}"
-  run_once: true
-  changed_when: false
-  when: not download_always_pull
-
-- name: Check the local digest sha256 corresponds to the given image tag
-  assert:
-    that: "{{ download.repo }}:{{ download.tag }} in docker_images.stdout.split(',')"
-  when:
-    - group_names | intersect(download.groups) | length
-    - not download_always_pull
-    - not pull_required
-    - pull_by_digest
-  tags:
-    - asserts

roles/download/tasks/sync_container.yml

@@ -1,141 +1,37 @@
 ---
-- name: container_download | Make download decision if pull is required by tag or sha256
-  include: set_docker_image_facts.yml
-  when:
-    - download.enabled
-    - download.container
-  tags:
-    - facts
-
-- name: container_download | Set file name of container tarballs
-  set_fact:
-    fname: "{{ local_release_dir }}/containers/{{ download.repo|regex_replace('/|\0|:', '_') }}:{{ download.tag|default(download.sha256)|regex_replace('/|\0|:', '_') }}.tar"
-  run_once: true
-  when:
-    - download.enabled
-    - download.container
-    - download_run_once
-  tags:
-    - facts
-
-- name: "container_download | Set default value for 'container_changed' to false"
-  set_fact:
-    container_changed: "{{ pull_required|default(false) }}"
-  when:
-    - download.enabled
-    - download.container
-    - download_run_once
-
-- name: "container_download | Update the 'container_changed' fact"
-  set_fact:
-    container_changed: "{{ pull_required|default(false) or not 'up to date' in pull_task_result.stdout }}"
-  when:
-    - download.enabled
-    - download.container
-    - download_run_once
-    - pull_required|default(download_always_pull)
-  run_once: "{{ download_run_once }}"
-  tags:
-    - facts
-
-- name: container_download | Stat saved container image
-  stat:
-    path: "{{ fname }}"
-  register: img
-  changed_when: false
-  delegate_to: "{{ download_delegate }}"
-  delegate_facts: no
-  become: false
-  run_once: true
-  when:
-    - download.enabled
-    - download.container
-    - download_run_once
-    - any_pull_required | default(download_always_pull)
-  tags:
+- block:
+  - name: sync_container | Gather information about the current image (how to download, is it cached etc.)
+    import_tasks: set_container_facts.yml
+    tags:
     - facts
 
-- name: container_download | save container images
-  shell: "{{ docker_bin_dir }}/docker save {{ pull_args }} | gzip -{{ download_compress }} > {{ fname }}"
-  delegate_to: "{{ download_delegate }}"
-  delegate_facts: no
-  register: saved
-  failed_when: saved.stderr
-  when:
-    - download.enabled
-    - download.container
-    - download_run_once
-    - any_pull_required | default(download_always_pull)
-    - (ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"] or download_delegate == "localhost")
-    - (container_changed or not img.stat.exists)
-
-- name: container_download | create container images directory on ansible host
-  file:
-    state: directory
-    path: "{{ fname | dirname }}"
-  delegate_to: localhost
-  delegate_facts: no
-  run_once: true
-  become: false
-  when:
-    - download.enabled
-    - download.container
-    - download_run_once
-    - any_pull_required | default(download_always_pull)
+  - name: sync_container | Upload container image to node
+    synchronize:
+      src: "{{ image_path_cached }}"
+      dest: "{{ image_path_final }}"
+      use_ssh_args: "{{ has_bastion | default(false) }}"
+      mode: push
+    delegate_facts: no
+    register: get_task
+    become: true
+    until: get_task is succeeded
+    retries: 4
+    delay: "{{ retry_stagger | random + 3 }}"
+    when:
     - ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"]
-    - inventory_hostname == download_delegate
-    - download_delegate != "localhost"
-    - saved.changed
 
-- name: container_download | copy container images to ansible host
-  synchronize:
-    src: "{{ fname }}"
-    dest: "{{ fname }}"
-    use_ssh_args: "{{ has_bastion | default(false) }}"
-    mode: pull
-    private_key: "{{ ansible_ssh_private_key_file }}"
-  become: false
-  when:
-    - download.enabled
-    - download.container
-    - download_run_once
+  - name: sync_container | Load container image into docker
+    shell: "{{ docker_bin_dir }}/docker load < {{ image_path_final }}"
+    when:
     - ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"]
-    - inventory_hostname == download_delegate
-    - download_delegate != "localhost"
-    - saved.changed
 
-- name: container_download | upload container images to nodes
-  synchronize:
-    src: "{{ fname }}"
-    dest: "{{ fname }}"
-    use_ssh_args: "{{ has_bastion | default(false) }}"
-    mode: push
-  become: true
-  register: get_task
-  until: get_task is succeeded
-  retries: 4
-  delay: "{{ retry_stagger | random + 3 }}"
-  when:
-    - download.enabled
-    - download.container
-    - download_run_once
-    - pull_required|default(download_always_pull)
-    - (ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"] and
-      inventory_hostname != download_delegate or
-      download_delegate == "localhost")
-  tags:
-    - upload
-    - upgrade
+  - name: sync_container | Remove container image from cache
+    file:
+      state: absent
+      path: "{{ image_path_final }}"
+    when:
+    - not download_keep_remote_cache
+    - ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"]
 
-- name: container_download | load container images
-  shell: "{{ docker_bin_dir }}/docker load < {{ fname }}"
-  when:
-    - download.enabled
-    - download.container
-    - download_run_once
-    - pull_required|default(download_always_pull)
-    - (ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"] and
-      inventory_hostname != download_delegate or download_delegate == "localhost")
   tags:
-    - upload
-    - upgrade
+  - upload

roles/download/tasks/sync_file.yml

@@ -1,53 +1,45 @@
 ---
-- name: file_download | create local download destination directory
-  file:
-    path: "{{ download.dest|dirname }}"
-    state: directory
-    recurse: yes
-    mode: 0755
-  delegate_to: localhost
-  become: false
-  run_once: true
-  when:
-    - download_delegate != "localhost"
-    - download_run_once
-    - download.enabled
-    - download.file
+- block:
+  - name: sync_file | Starting file sync of file
+    debug:
+      msg: "Starting file sync of file: {{ download.dest }}"
 
-- name: file_download | copy file to ansible host
-  synchronize:
-    src: "{{ download.dest }}"
-    dest: "{{ download.dest }}"
-    use_ssh_args: "{{ has_bastion | default(false) }}"
-    mode: pull
-  run_once: true
-  become: false
-  when:
-    - download.enabled
-    - download.file
-    - download_run_once
+  - name: download_file | Set pathname of cached file
+    set_fact:
+      file_path_cached: "{{ download_cache_dir }}/{{ download.dest | regex_replace('^\\/', '') }}"
+    tags:
+    - facts
+
+  - name: sync_file | Create dest directory on node
+    file:
+      path: "{{ download.dest | dirname }}"
+      owner: "{{ download.owner | default(omit) }}"
+      mode: 0755
+      state: directory
+      recurse: yes
+
+  - name: sync_file | Upload file images to node
+    synchronize:
+      src: "{{ file_path_cached }}"
+      dest: "{{ download.dest }}"
+      use_ssh_args: "{{ has_bastion | default(false) }}"
+      mode: push
+    become: true
+    register: get_task
+    until: get_task is succeeded
+    retries: 4
+    delay: "{{ retry_stagger | random + 3 }}"
+    when:
     - ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"]
-    - inventory_hostname == download_delegate
-    - download_delegate != "localhost"
 
-- name: file_download | upload file to nodes
-  synchronize:
-    src: "{{ download.dest }}"
-    dest: "{{ download.dest }}"
-    use_ssh_args: "{{ has_bastion | default(false) }}"
-    mode: push
-  become: true
-  register: get_task
-  until: get_task is succeeded
-  retries: 4
-  delay: "{{ retry_stagger | random + 3 }}"
-  when:
-    - download.enabled
-    - download.file
-    - download_run_once
-    - (ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"] and
-      inventory_hostname != download_delegate or
-      download_delegate == "localhost")
+  - name: sync_file | Set mode and owner
+    file:
+      path: "{{ download.dest }}"
+      mode: "{{ download.mode | default(omit) }}"
+      owner: "{{ download.owner | default(omit) }}"
+
+  - name: sync_file | Extract file archives
+    include_tasks: "extract_file.yml"
+
   tags:
-    - upload
-    - upgrade
+  - upload