You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

123 lines
3.9 KiB

Added file and container image caching (#4828) * File and container image downloads are now cached localy, so that repeated vagrant up/down runs do not trigger downloading of those files. This is especially useful on laptops with kubernetes runnig locally on vm's. The total size of the cache, after an ansible run, is currently around 800MB, so bandwidth (=time) savings can be quite significant. * When download_run_once is false, the default is still not to cache, but setting download_force_cache will still enable caching. * The local cache location can be set with download_cache_dir and defaults to /tmp/kubernetes_cache * A local docker instance is no longer required to cache docker images; Images are cached to file. A local docker instance is still required, though, if you wish to download images on localhost. * Fixed a FIXME, wher the argument was that delegate_to doesn't play nice with omit. That is a correct observation and the fix is to use default(inventory_host) instead of default(omit). See ansible/ansible#26009 * Removed "Register docker images info" task from download_container and set_docker_image_facts because it was faulty and unused. * Removed redundant when:download.{container,enabled,run_once} conditions from {sync,download}_container.yml * All features of commit d6fd0d2acaec9f53e75d82db30411f96a5bf2cc9 by Timoses <timosesu@gmail.com>, merged May 1st 2019, are included in this patch. Not all code was included verbatim, but each feature of that commit was checked to be working in this patch. One notable change: The actual downloading of the kubeadm images was moved to {download,sync)_container, to enable caching. Note 1: I considered splitting this patch, but most changes that are not directly related to caching, are a pleasant by-product of implementing the caching code, so splitting would be impractical. Note 2: I have my doubts about the usefulness of the upload, download and upgrade tags in the download role. Must they remain or can they be removed? If anybody knows, then please speak up.
5 years ago
Added file and container image caching (#4828) * File and container image downloads are now cached localy, so that repeated vagrant up/down runs do not trigger downloading of those files. This is especially useful on laptops with kubernetes runnig locally on vm's. The total size of the cache, after an ansible run, is currently around 800MB, so bandwidth (=time) savings can be quite significant. * When download_run_once is false, the default is still not to cache, but setting download_force_cache will still enable caching. * The local cache location can be set with download_cache_dir and defaults to /tmp/kubernetes_cache * A local docker instance is no longer required to cache docker images; Images are cached to file. A local docker instance is still required, though, if you wish to download images on localhost. * Fixed a FIXME, wher the argument was that delegate_to doesn't play nice with omit. That is a correct observation and the fix is to use default(inventory_host) instead of default(omit). See ansible/ansible#26009 * Removed "Register docker images info" task from download_container and set_docker_image_facts because it was faulty and unused. * Removed redundant when:download.{container,enabled,run_once} conditions from {sync,download}_container.yml * All features of commit d6fd0d2acaec9f53e75d82db30411f96a5bf2cc9 by Timoses <timosesu@gmail.com>, merged May 1st 2019, are included in this patch. Not all code was included verbatim, but each feature of that commit was checked to be working in this patch. One notable change: The actual downloading of the kubeadm images was moved to {download,sync)_container, to enable caching. Note 1: I considered splitting this patch, but most changes that are not directly related to caching, are a pleasant by-product of implementing the caching code, so splitting would be impractical. Note 2: I have my doubts about the usefulness of the upload, download and upgrade tags in the download role. Must they remain or can they be removed? If anybody knows, then please speak up.
5 years ago
Added file and container image caching (#4828) * File and container image downloads are now cached localy, so that repeated vagrant up/down runs do not trigger downloading of those files. This is especially useful on laptops with kubernetes runnig locally on vm's. The total size of the cache, after an ansible run, is currently around 800MB, so bandwidth (=time) savings can be quite significant. * When download_run_once is false, the default is still not to cache, but setting download_force_cache will still enable caching. * The local cache location can be set with download_cache_dir and defaults to /tmp/kubernetes_cache * A local docker instance is no longer required to cache docker images; Images are cached to file. A local docker instance is still required, though, if you wish to download images on localhost. * Fixed a FIXME, wher the argument was that delegate_to doesn't play nice with omit. That is a correct observation and the fix is to use default(inventory_host) instead of default(omit). See ansible/ansible#26009 * Removed "Register docker images info" task from download_container and set_docker_image_facts because it was faulty and unused. * Removed redundant when:download.{container,enabled,run_once} conditions from {sync,download}_container.yml * All features of commit d6fd0d2acaec9f53e75d82db30411f96a5bf2cc9 by Timoses <timosesu@gmail.com>, merged May 1st 2019, are included in this patch. Not all code was included verbatim, but each feature of that commit was checked to be working in this patch. One notable change: The actual downloading of the kubeadm images was moved to {download,sync)_container, to enable caching. Note 1: I considered splitting this patch, but most changes that are not directly related to caching, are a pleasant by-product of implementing the caching code, so splitting would be impractical. Note 2: I have my doubts about the usefulness of the upload, download and upgrade tags in the download role. Must they remain or can they be removed? If anybody knows, then please speak up.
5 years ago
Added file and container image caching (#4828) * File and container image downloads are now cached localy, so that repeated vagrant up/down runs do not trigger downloading of those files. This is especially useful on laptops with kubernetes runnig locally on vm's. The total size of the cache, after an ansible run, is currently around 800MB, so bandwidth (=time) savings can be quite significant. * When download_run_once is false, the default is still not to cache, but setting download_force_cache will still enable caching. * The local cache location can be set with download_cache_dir and defaults to /tmp/kubernetes_cache * A local docker instance is no longer required to cache docker images; Images are cached to file. A local docker instance is still required, though, if you wish to download images on localhost. * Fixed a FIXME, wher the argument was that delegate_to doesn't play nice with omit. That is a correct observation and the fix is to use default(inventory_host) instead of default(omit). See ansible/ansible#26009 * Removed "Register docker images info" task from download_container and set_docker_image_facts because it was faulty and unused. * Removed redundant when:download.{container,enabled,run_once} conditions from {sync,download}_container.yml * All features of commit d6fd0d2acaec9f53e75d82db30411f96a5bf2cc9 by Timoses <timosesu@gmail.com>, merged May 1st 2019, are included in this patch. Not all code was included verbatim, but each feature of that commit was checked to be working in this patch. One notable change: The actual downloading of the kubeadm images was moved to {download,sync)_container, to enable caching. Note 1: I considered splitting this patch, but most changes that are not directly related to caching, are a pleasant by-product of implementing the caching code, so splitting would be impractical. Note 2: I have my doubts about the usefulness of the upload, download and upgrade tags in the download role. Must they remain or can they be removed? If anybody knows, then please speak up.
5 years ago
Added file and container image caching (#4828) * File and container image downloads are now cached localy, so that repeated vagrant up/down runs do not trigger downloading of those files. This is especially useful on laptops with kubernetes runnig locally on vm's. The total size of the cache, after an ansible run, is currently around 800MB, so bandwidth (=time) savings can be quite significant. * When download_run_once is false, the default is still not to cache, but setting download_force_cache will still enable caching. * The local cache location can be set with download_cache_dir and defaults to /tmp/kubernetes_cache * A local docker instance is no longer required to cache docker images; Images are cached to file. A local docker instance is still required, though, if you wish to download images on localhost. * Fixed a FIXME, wher the argument was that delegate_to doesn't play nice with omit. That is a correct observation and the fix is to use default(inventory_host) instead of default(omit). See ansible/ansible#26009 * Removed "Register docker images info" task from download_container and set_docker_image_facts because it was faulty and unused. * Removed redundant when:download.{container,enabled,run_once} conditions from {sync,download}_container.yml * All features of commit d6fd0d2acaec9f53e75d82db30411f96a5bf2cc9 by Timoses <timosesu@gmail.com>, merged May 1st 2019, are included in this patch. Not all code was included verbatim, but each feature of that commit was checked to be working in this patch. One notable change: The actual downloading of the kubeadm images was moved to {download,sync)_container, to enable caching. Note 1: I considered splitting this patch, but most changes that are not directly related to caching, are a pleasant by-product of implementing the caching code, so splitting would be impractical. Note 2: I have my doubts about the usefulness of the upload, download and upgrade tags in the download role. Must they remain or can they be removed? If anybody knows, then please speak up.
5 years ago
  1. ---
  2. - block:
  3. - name: download_file | Starting dowload of file
  4. debug:
  5. msg: "{{ download.url }}"
  6. run_once: "{{ download_run_once }}"
  7. - name: download_file | Set pathname of cached file
  8. set_fact:
  9. file_path_cached: "{{ download_cache_dir }}/{{ download.dest | regex_replace('^\\/', '') }}"
  10. tags:
  11. - facts
  12. - name: download_file | Create dest directory on node
  13. file:
  14. path: "{{ download.dest | dirname }}"
  15. owner: "{{ download.owner | default(omit) }}"
  16. mode: 0755
  17. state: directory
  18. recurse: yes
  19. - name: download_file | Create local cache directory
  20. file:
  21. path: "{{ file_path_cached | dirname }}"
  22. state: directory
  23. recurse: yes
  24. delegate_to: localhost
  25. delegate_facts: false
  26. run_once: true
  27. become: false
  28. tags:
  29. - localhost
  30. - name: download_file | Check if file is available in cache
  31. stat:
  32. path: "{{ file_path_cached }}"
  33. register: cache_file
  34. run_once: true
  35. changed_when: false
  36. delegate_to: localhost
  37. delegate_facts: no
  38. become: false
  39. when:
  40. - download_force_cache
  41. tags:
  42. - facts
  43. - name: download_file | Set file_is_cached fact based on previous task
  44. set_fact:
  45. file_is_cached: "{{ cache_file.stat.exists | default(false) }}"
  46. when:
  47. - download_force_cache
  48. tags:
  49. - facts
  50. - name: download_file | Copy file from cache to nodes, if it is available
  51. synchronize:
  52. src: "{{ file_path_cached }}"
  53. dest: "{{ download.dest }}"
  54. use_ssh_args: "{{ has_bastion | default(false) }}"
  55. mode: push
  56. run_once: "{{ download_run_once }}"
  57. register: get_task
  58. until: get_task is succeeded
  59. delay: "{{ retry_stagger | random + 3 }}"
  60. retries: 4
  61. when:
  62. - download_force_cache
  63. - file_is_cached
  64. - ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"]
  65. - name: download_file | Set mode and owner
  66. file:
  67. path: "{{ download.dest }}"
  68. mode: "{{ download.mode | default(omit) }}"
  69. owner: "{{ download.owner | default(omit) }}"
  70. run_once: "{{ download_run_once }}"
  71. when:
  72. - download_force_cache
  73. - file_is_cached
  74. - ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"]
  75. # This must always be called, to check if the checksum matches. On no-match the file is re-downloaded.
  76. - name: download_file | Download item
  77. get_url:
  78. url: "{{ download.url }}"
  79. dest: "{{ file_path_cached if download_localhost else download.dest }}"
  80. owner: "{{ omit if download_localhost else (download.owner | default(omit)) }}"
  81. mode: "{{ omit if download_localhost else (download.mode | default(omit)) }}"
  82. checksum: "{{ 'sha256:' + download.sha256 if download.sha256 or omit }}"
  83. validate_certs: "{{ download_validate_certs }}"
  84. url_username: "{{ download.username | default(omit) }}"
  85. url_password: "{{ download.password | default(omit) }}"
  86. force_basic_auth: "{{ download.force_basic_auth | default(omit) }}"
  87. delegate_to: "{{ download_delegate if download_run_once else inventory_hostname }}"
  88. run_once: "{{ download_run_once }}"
  89. register: get_url_result
  90. become: "{{ not download_localhost }}"
  91. until: "'OK' in get_url_result.msg or 'file already exists' in get_url_result.msg"
  92. retries: 4
  93. delay: "{{ retry_stagger | default(5) }}"
  94. - name: "download_file | Extract file archives"
  95. include_tasks: "extract_file.yml"
  96. when:
  97. - not download_localhost
  98. - name: download_file | Copy file back to ansible host file cache
  99. synchronize:
  100. src: "{{ download.dest }}"
  101. dest: "{{ file_path_cached }}"
  102. use_ssh_args: "{{ has_bastion | default(false) }}"
  103. mode: pull
  104. run_once: true
  105. when:
  106. - download_force_cache
  107. - not file_is_cached or get_url_result.changed
  108. - download_delegate == inventory_hostname
  109. - not (download_run_once and download_delegate == 'localhost')
  110. - ansible_os_family not in ["CoreOS", "Container Linux by CoreOS"]
  111. tags:
  112. - download