You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

125 lines
4.3 KiB

Added file and container image caching (#4828) * File and container image downloads are now cached localy, so that repeated vagrant up/down runs do not trigger downloading of those files. This is especially useful on laptops with kubernetes runnig locally on vm's. The total size of the cache, after an ansible run, is currently around 800MB, so bandwidth (=time) savings can be quite significant. * When download_run_once is false, the default is still not to cache, but setting download_force_cache will still enable caching. * The local cache location can be set with download_cache_dir and defaults to /tmp/kubernetes_cache * A local docker instance is no longer required to cache docker images; Images are cached to file. A local docker instance is still required, though, if you wish to download images on localhost. * Fixed a FIXME, wher the argument was that delegate_to doesn't play nice with omit. That is a correct observation and the fix is to use default(inventory_host) instead of default(omit). See ansible/ansible#26009 * Removed "Register docker images info" task from download_container and set_docker_image_facts because it was faulty and unused. * Removed redundant when:download.{container,enabled,run_once} conditions from {sync,download}_container.yml * All features of commit d6fd0d2acaec9f53e75d82db30411f96a5bf2cc9 by Timoses <timosesu@gmail.com>, merged May 1st 2019, are included in this patch. Not all code was included verbatim, but each feature of that commit was checked to be working in this patch. One notable change: The actual downloading of the kubeadm images was moved to {download,sync)_container, to enable caching. Note 1: I considered splitting this patch, but most changes that are not directly related to caching, are a pleasant by-product of implementing the caching code, so splitting would be impractical. Note 2: I have my doubts about the usefulness of the upload, download and upgrade tags in the download role. Must they remain or can they be removed? If anybody knows, then please speak up.
5 years ago
Added file and container image caching (#4828) * File and container image downloads are now cached localy, so that repeated vagrant up/down runs do not trigger downloading of those files. This is especially useful on laptops with kubernetes runnig locally on vm's. The total size of the cache, after an ansible run, is currently around 800MB, so bandwidth (=time) savings can be quite significant. * When download_run_once is false, the default is still not to cache, but setting download_force_cache will still enable caching. * The local cache location can be set with download_cache_dir and defaults to /tmp/kubernetes_cache * A local docker instance is no longer required to cache docker images; Images are cached to file. A local docker instance is still required, though, if you wish to download images on localhost. * Fixed a FIXME, wher the argument was that delegate_to doesn't play nice with omit. That is a correct observation and the fix is to use default(inventory_host) instead of default(omit). See ansible/ansible#26009 * Removed "Register docker images info" task from download_container and set_docker_image_facts because it was faulty and unused. * Removed redundant when:download.{container,enabled,run_once} conditions from {sync,download}_container.yml * All features of commit d6fd0d2acaec9f53e75d82db30411f96a5bf2cc9 by Timoses <timosesu@gmail.com>, merged May 1st 2019, are included in this patch. Not all code was included verbatim, but each feature of that commit was checked to be working in this patch. One notable change: The actual downloading of the kubeadm images was moved to {download,sync)_container, to enable caching. Note 1: I considered splitting this patch, but most changes that are not directly related to caching, are a pleasant by-product of implementing the caching code, so splitting would be impractical. Note 2: I have my doubts about the usefulness of the upload, download and upgrade tags in the download role. Must they remain or can they be removed? If anybody knows, then please speak up.
5 years ago
Added file and container image caching (#4828) * File and container image downloads are now cached localy, so that repeated vagrant up/down runs do not trigger downloading of those files. This is especially useful on laptops with kubernetes runnig locally on vm's. The total size of the cache, after an ansible run, is currently around 800MB, so bandwidth (=time) savings can be quite significant. * When download_run_once is false, the default is still not to cache, but setting download_force_cache will still enable caching. * The local cache location can be set with download_cache_dir and defaults to /tmp/kubernetes_cache * A local docker instance is no longer required to cache docker images; Images are cached to file. A local docker instance is still required, though, if you wish to download images on localhost. * Fixed a FIXME, wher the argument was that delegate_to doesn't play nice with omit. That is a correct observation and the fix is to use default(inventory_host) instead of default(omit). See ansible/ansible#26009 * Removed "Register docker images info" task from download_container and set_docker_image_facts because it was faulty and unused. * Removed redundant when:download.{container,enabled,run_once} conditions from {sync,download}_container.yml * All features of commit d6fd0d2acaec9f53e75d82db30411f96a5bf2cc9 by Timoses <timosesu@gmail.com>, merged May 1st 2019, are included in this patch. Not all code was included verbatim, but each feature of that commit was checked to be working in this patch. One notable change: The actual downloading of the kubeadm images was moved to {download,sync)_container, to enable caching. Note 1: I considered splitting this patch, but most changes that are not directly related to caching, are a pleasant by-product of implementing the caching code, so splitting would be impractical. Note 2: I have my doubts about the usefulness of the upload, download and upgrade tags in the download role. Must they remain or can they be removed? If anybody knows, then please speak up.
5 years ago
Added file and container image caching (#4828) * File and container image downloads are now cached localy, so that repeated vagrant up/down runs do not trigger downloading of those files. This is especially useful on laptops with kubernetes runnig locally on vm's. The total size of the cache, after an ansible run, is currently around 800MB, so bandwidth (=time) savings can be quite significant. * When download_run_once is false, the default is still not to cache, but setting download_force_cache will still enable caching. * The local cache location can be set with download_cache_dir and defaults to /tmp/kubernetes_cache * A local docker instance is no longer required to cache docker images; Images are cached to file. A local docker instance is still required, though, if you wish to download images on localhost. * Fixed a FIXME, wher the argument was that delegate_to doesn't play nice with omit. That is a correct observation and the fix is to use default(inventory_host) instead of default(omit). See ansible/ansible#26009 * Removed "Register docker images info" task from download_container and set_docker_image_facts because it was faulty and unused. * Removed redundant when:download.{container,enabled,run_once} conditions from {sync,download}_container.yml * All features of commit d6fd0d2acaec9f53e75d82db30411f96a5bf2cc9 by Timoses <timosesu@gmail.com>, merged May 1st 2019, are included in this patch. Not all code was included verbatim, but each feature of that commit was checked to be working in this patch. One notable change: The actual downloading of the kubeadm images was moved to {download,sync)_container, to enable caching. Note 1: I considered splitting this patch, but most changes that are not directly related to caching, are a pleasant by-product of implementing the caching code, so splitting would be impractical. Note 2: I have my doubts about the usefulness of the upload, download and upgrade tags in the download role. Must they remain or can they be removed? If anybody knows, then please speak up.
5 years ago
Added file and container image caching (#4828) * File and container image downloads are now cached localy, so that repeated vagrant up/down runs do not trigger downloading of those files. This is especially useful on laptops with kubernetes runnig locally on vm's. The total size of the cache, after an ansible run, is currently around 800MB, so bandwidth (=time) savings can be quite significant. * When download_run_once is false, the default is still not to cache, but setting download_force_cache will still enable caching. * The local cache location can be set with download_cache_dir and defaults to /tmp/kubernetes_cache * A local docker instance is no longer required to cache docker images; Images are cached to file. A local docker instance is still required, though, if you wish to download images on localhost. * Fixed a FIXME, wher the argument was that delegate_to doesn't play nice with omit. That is a correct observation and the fix is to use default(inventory_host) instead of default(omit). See ansible/ansible#26009 * Removed "Register docker images info" task from download_container and set_docker_image_facts because it was faulty and unused. * Removed redundant when:download.{container,enabled,run_once} conditions from {sync,download}_container.yml * All features of commit d6fd0d2acaec9f53e75d82db30411f96a5bf2cc9 by Timoses <timosesu@gmail.com>, merged May 1st 2019, are included in this patch. Not all code was included verbatim, but each feature of that commit was checked to be working in this patch. One notable change: The actual downloading of the kubeadm images was moved to {download,sync)_container, to enable caching. Note 1: I considered splitting this patch, but most changes that are not directly related to caching, are a pleasant by-product of implementing the caching code, so splitting would be impractical. Note 2: I have my doubts about the usefulness of the upload, download and upgrade tags in the download role. Must they remain or can they be removed? If anybody knows, then please speak up.
5 years ago
Upgrade ansible (#10190) * project: update all dependencies including ansible Upgrade to ansible 7.x and ansible-core 2.14.x. There seems to be issue with ansible 8/ansible-core 2.15 so we remain on those versions for now. It's quite a big bump already anyway. Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * tests: install aws galaxy collection Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * ansible-lint: disable various rules after ansible upgrade Temporarily disable a bunch of linting action following ansible upgrade. Those should be taken care of separately. Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: resolve deprecated-module ansible-lint error Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: resolve no-free-form ansible-lint error Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: resolve schema[meta] ansible-lint error Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: resolve schema[playbook] ansible-lint error Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: resolve schema[tasks] ansible-lint error Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: resolve risky-file-permissions ansible-lint error Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: resolve risky-shell-pipe ansible-lint error Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: remove deprecated warn args Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: use fqcn for non builtin tasks Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: resolve syntax-check[missing-file] for contrib playbook Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: use arithmetic inside jinja to fix ansible 6 upgrade Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> --------- Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch>
1 year ago
Upgrade ansible (#10190) * project: update all dependencies including ansible Upgrade to ansible 7.x and ansible-core 2.14.x. There seems to be issue with ansible 8/ansible-core 2.15 so we remain on those versions for now. It's quite a big bump already anyway. Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * tests: install aws galaxy collection Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * ansible-lint: disable various rules after ansible upgrade Temporarily disable a bunch of linting action following ansible upgrade. Those should be taken care of separately. Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: resolve deprecated-module ansible-lint error Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: resolve no-free-form ansible-lint error Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: resolve schema[meta] ansible-lint error Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: resolve schema[playbook] ansible-lint error Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: resolve schema[tasks] ansible-lint error Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: resolve risky-file-permissions ansible-lint error Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: resolve risky-shell-pipe ansible-lint error Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: remove deprecated warn args Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: use fqcn for non builtin tasks Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: resolve syntax-check[missing-file] for contrib playbook Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> * project: use arithmetic inside jinja to fix ansible 6 upgrade Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch> --------- Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch>
1 year ago
Added file and container image caching (#4828) * File and container image downloads are now cached localy, so that repeated vagrant up/down runs do not trigger downloading of those files. This is especially useful on laptops with kubernetes runnig locally on vm's. The total size of the cache, after an ansible run, is currently around 800MB, so bandwidth (=time) savings can be quite significant. * When download_run_once is false, the default is still not to cache, but setting download_force_cache will still enable caching. * The local cache location can be set with download_cache_dir and defaults to /tmp/kubernetes_cache * A local docker instance is no longer required to cache docker images; Images are cached to file. A local docker instance is still required, though, if you wish to download images on localhost. * Fixed a FIXME, wher the argument was that delegate_to doesn't play nice with omit. That is a correct observation and the fix is to use default(inventory_host) instead of default(omit). See ansible/ansible#26009 * Removed "Register docker images info" task from download_container and set_docker_image_facts because it was faulty and unused. * Removed redundant when:download.{container,enabled,run_once} conditions from {sync,download}_container.yml * All features of commit d6fd0d2acaec9f53e75d82db30411f96a5bf2cc9 by Timoses <timosesu@gmail.com>, merged May 1st 2019, are included in this patch. Not all code was included verbatim, but each feature of that commit was checked to be working in this patch. One notable change: The actual downloading of the kubeadm images was moved to {download,sync)_container, to enable caching. Note 1: I considered splitting this patch, but most changes that are not directly related to caching, are a pleasant by-product of implementing the caching code, so splitting would be impractical. Note 2: I have my doubts about the usefulness of the upload, download and upgrade tags in the download role. Must they remain or can they be removed? If anybody knows, then please speak up.
5 years ago
  1. ---
  2. - tags:
  3. - download
  4. block:
  5. - name: Set default values for flag variables
  6. set_fact:
  7. image_is_cached: false
  8. image_changed: false
  9. pull_required: "{{ download_always_pull }}"
  10. tags:
  11. - facts
  12. - name: Download_container | Set a few facts
  13. import_tasks: set_container_facts.yml
  14. tags:
  15. - facts
  16. - name: Download_container | Prepare container download
  17. include_tasks: check_pull_required.yml
  18. when:
  19. - not download_always_pull
  20. - debug: # noqa name[missing]
  21. msg: "Pull {{ image_reponame }} required is: {{ pull_required }}"
  22. - name: Download_container | Determine if image is in cache
  23. stat:
  24. path: "{{ image_path_cached }}"
  25. get_attributes: false
  26. get_checksum: false
  27. get_mime: false
  28. delegate_to: localhost
  29. connection: local
  30. delegate_facts: false
  31. register: cache_image
  32. changed_when: false
  33. become: false
  34. when:
  35. - download_force_cache
  36. - name: Download_container | Set fact indicating if image is in cache
  37. set_fact:
  38. image_is_cached: "{{ cache_image.stat.exists }}"
  39. tags:
  40. - facts
  41. when:
  42. - download_force_cache
  43. - name: Stop if image not in cache on ansible host when download_force_cache=true
  44. assert:
  45. that: image_is_cached
  46. msg: "Image cache file {{ image_path_cached }} not found for {{ image_reponame }} on localhost"
  47. when:
  48. - download_force_cache
  49. - not download_run_once
  50. - name: Download_container | Download image if required
  51. command: "{{ image_pull_command_on_localhost if download_localhost else image_pull_command }} {{ image_reponame }}"
  52. delegate_to: "{{ download_delegate if download_run_once else inventory_hostname }}"
  53. delegate_facts: true
  54. run_once: "{{ download_run_once }}"
  55. register: pull_task_result
  56. until: pull_task_result is succeeded
  57. delay: "{{ retry_stagger | random + 3 }}"
  58. retries: "{{ download_retries }}"
  59. become: "{{ user_can_become_root | default(false) or not download_localhost }}"
  60. environment: "{{ proxy_env if container_manager == 'containerd' else omit }}"
  61. when:
  62. - pull_required or download_run_once
  63. - not image_is_cached
  64. - name: Download_container | Save and compress image
  65. shell: "{{ image_save_command_on_localhost if download_localhost else image_save_command }}" # noqa command-instead-of-shell - image_save_command_on_localhost contains a pipe, therefore requires shell
  66. delegate_to: "{{ download_delegate }}"
  67. delegate_facts: false
  68. register: container_save_status
  69. failed_when: container_save_status.stderr
  70. run_once: true
  71. become: "{{ user_can_become_root | default(false) or not download_localhost }}"
  72. when:
  73. - not image_is_cached
  74. - download_run_once
  75. - name: Download_container | Copy image to ansible host cache
  76. ansible.posix.synchronize:
  77. src: "{{ image_path_final }}"
  78. dest: "{{ image_path_cached }}"
  79. use_ssh_args: true
  80. mode: pull
  81. when:
  82. - not image_is_cached
  83. - download_run_once
  84. - not download_localhost
  85. - download_delegate == inventory_hostname
  86. - name: Download_container | Upload image to node if it is cached
  87. ansible.posix.synchronize:
  88. src: "{{ image_path_cached }}"
  89. dest: "{{ image_path_final }}"
  90. use_ssh_args: true
  91. mode: push
  92. delegate_facts: false
  93. register: upload_image
  94. failed_when: not upload_image
  95. until: upload_image is succeeded
  96. retries: "{{ download_retries }}"
  97. delay: "{{ retry_stagger | random + 3 }}"
  98. when:
  99. - pull_required
  100. - download_force_cache
  101. - name: Download_container | Load image into the local container registry
  102. shell: "{{ image_load_command }}" # noqa command-instead-of-shell - image_load_command uses pipes, therefore requires shell
  103. register: container_load_status
  104. failed_when: container_load_status is failed
  105. when:
  106. - pull_required
  107. - download_force_cache
  108. - name: Download_container | Remove container image from cache
  109. file:
  110. state: absent
  111. path: "{{ image_path_final }}"
  112. when:
  113. - not download_keep_remote_cache