Ansible cluster installation #3

ventifus · 2024-09-27T23:48:15Z

Migrating Azure/ARO-RP#3843 to aro-e2e. Refactored to move the roles into an Ansible collection azureredhatopenshift.cluster.

What this PR does / why we need it:

This PR leverages Ansible to automate cluster creation in predefined configurations. These configurations are intended to exercise ARO features in a repeatable manner. For example, we want to make sure that various complex cluster configurations continue to work in production:

UserDefinedRouting with blackhole route
Disk encryption at host
Disk encryption with bring-your-own-key
Bring-your-own-NSG (TODO)

Also implemented is the ability to upgrade through a sequence of versions. See ansible/inventory/upgrades.yaml and ansible/inventory/upgrades-candidate.yamlfor examples.

Refer to ansible/README.md for details how to run the Ansible playbooks via make ansible-image and make cluster targets.

For development, this also adds lint-ansible and test-ansible targets. Lint requires ansible-lint to be installed. Linting also runs as part of the container image build. test-ansible runs inside an aro-ansible container and does not need ansible installed.

lranjbar · 2024-10-03T18:20:00Z

lgtm

ventifus · 2024-10-03T21:22:14Z

/assign sudobrendan

lranjbar · 2024-10-30T15:05:22Z

/lgtm

lranjbar · 2024-10-30T15:50:23Z

/lgtm

s-fairchild

LGTM, I left some questions. Namely just asking about code that's commented out.
Can we clean this up by removing the unused code? If it's intended for implementation later, I recommend creating a new branch based on this branch, and deleting it from here.

s-fairchild · 2024-10-30T16:26:41Z

ansible_collections/azureredhatopenshift/cluster/meta/runtime.yml

+# Content that Ansible needs to load from another location or that has
+# been deprecated/removed
+# plugin_routing:
+#   action:
+#     redirected_plugin_name:
+#       redirect: ns.col.new_location
+#     deprecated_plugin_name:
+#       deprecation:
+#         removal_version: "4.0.0"
+#         warning_text: |
+#           See the porting guide on how to update your playbook to
+#           use ns.col.another_plugin instead.
+#     removed_plugin_name:
+#       tombstone:
+#         removal_version: "2.0.0"
+#         warning_text: |
+#           See the porting guide on how to update your playbook to
+#           use ns.col.another_plugin instead.
+#   become:
+#   cache:
+#   callback:
+#   cliconf:
+#   connection:
+#   doc_fragments:
+#   filter:
+#   httpapi:
+#   inventory:
+#   lookup:
+#   module_utils:
+#   modules:
+#   netconf:
+#   shell:
+#   strategy:
+#   terminal:
+#   test:
+#   vars:
+
+# Python import statements that Ansible needs to load from another location
+# import_redirection:
+#   ansible_collections.ns.col.plugins.module_utils.old_location:
+#     redirect: ansible_collections.ns.col.plugins.module_utils.new_location
+
+# Groups of actions/modules that take a common set of options
+# action_groups:
+#   group_name:
+#     - module1
+#     - module2


Should this be removed?

This came from the ansible project template, I've deleted it

s-fairchild · 2024-10-30T16:27:19Z

ansible_collections/azureredhatopenshift/cluster/roles/smoketest/tasks/new_project.yaml

+#  oc project default (this test assumes we are using the default ns)
+#  oc new-app --image=nginx && oc expose svc/nginx
+#  oc get deployment,pod,route,svc
+#  curl -I --silent http://<hostname you get from the route>
+
+# "Apply the following yml file:
+# apiVersion: v1
+# kind: PersistentVolumeClaim
+# metadata:
+#   name: azurefile-csi
+# spec:
+#   storageClassName: azurefile-csi
+#   accessModes:
+#     - ReadWriteOnce
+#   resources:
+#     requests:
+#       storage: 3Gi
+# ---
+# apiVersion: v1
+# kind: PersistentVolumeClaim
+# metadata:
+#   name: managed-csi
+# spec:
+#   storageClassName: managed-csi
+#   accessModes:
+#     - ReadWriteOnce
+#   resources:
+#     requests:
+#       storage: 3Gi
+# ---
+# apiVersion: v1
+# kind: PersistentVolumeClaim
+# metadata:
+#   name: managed-csi-encrypted-cmk
+# spec:
+#   storageClassName: managed-csi-encrypted-cmk
+#   accessModes:
+#     - ReadWriteOnce
+#   resources:
+#     requests:
+#       storage: 3Gi
+# "


Should this be removed?

This, and several similar comments were describing the manual steps from the smoketest spreadsheet that was getting automated.

s-fairchild · 2024-10-30T16:27:31Z

ansible_collections/azureredhatopenshift/cluster/roles/smoketest/tasks/new_project.yaml

+#       storage: 3Gi
+# "
+
+# TODO: Push the required images to the cluster's internal registry so it will work on disconnected clusters.


Is this a TODO for later, or in this PR?

Later, it's not needed immediately

s-fairchild · 2024-10-30T16:27:55Z

ansible_collections/azureredhatopenshift/cluster/tasks/create_aro_cluster.yaml

+    # - name: create_aro_cluster | Download azaroext
+    #   ansible.builtin.get_url:
+    #     url: https://arosvc.blob.core.windows.net/azext/aro-{{ AZAROEXT_VERSION }}-py2.py3-none-any.whl
+    #     dest: /tmp/aro-{{ AZAROEXT_VERSION }}-py2.py3-none-any.whl
+    #     mode: "0644"
+    #   delegate: "{{ delegation }}"
+    #   register: download_aroazext
+    # - name: create_aro_cluster | Install azaroext
+    #   when: download_aroazext is changed
+    #   ansible.builtin.command:
+    #     argv:
+    #       - az
+    #       - extension
+    #       - add
+    #       - -s="/tmp/aro-{{ AZAROEXT_VERSION }}-py2.py3-none-any.whl"


Should this be removed?

s-fairchild · 2024-10-30T16:28:11Z

ansible_collections/azureredhatopenshift/cluster/tasks/create_aro_cluster.yaml

+    #       - add
+    #       - -s="/tmp/aro-{{ AZAROEXT_VERSION }}-py2.py3-none-any.whl"
+
+# TODO: Create clusters with azure_rm_openshiftmanagedcluster if possible


Is this for this PR, or later?

Later; the ansible module azure_rm_openshiftmanagedcluster isn't at parity with our aro cli, we should work with the Azure Ansible team to get it updated.

s-fairchild · 2024-10-30T16:30:17Z

ansible_collections/azureredhatopenshift/cluster/tasks/create_vnet.yaml

+    # private_link_service_network_policies:
+    #     "{% if outbound_type is defined and outbound_type == 'UserDefinedRouting' %}Disabled{% else %}{{ omit }}{% endif %}"


Should this be removed?

s-fairchild · 2024-10-30T16:30:28Z

ansible_collections/azureredhatopenshift/cluster/tasks/delete_aro_cluster.yaml

+      # azure.azcollection.azure_rm_adserviceprincipal:
+      #   app_id: "{{ aro_ad_sp_info.id }}"
+      #   state: absent


Same as above

This is a todo for later

s-fairchild · 2024-10-30T16:30:38Z

ansible_collections/azureredhatopenshift/cluster/tasks/delete_aro_cluster.yaml

+      # azure.azcollection.azure_rm_adapplication:
+      #   app_id: "{{ aro_ad_sp_info.appId }}"
+      #   state: absent


Same as above

This is a todo for later

s-fairchild · 2024-10-30T16:30:52Z

ansible_collections/azureredhatopenshift/cluster/tasks/upgrade_cluster.yaml

+# - name: upgrade_cluster | Begin upgrade process
+#   loop: "{{ upgrade }}"
+#   when: upgrade is defined
+#   ansible.builtin.include_tasks:
+#     file: ../../tasks/upgrade_cluster.yaml


Same as above

This is documentation on how to include this particular task file.

s-fairchild · 2024-10-30T16:31:32Z

Makefile

+endif
+
+NO_CACHE ?= true
+PODMAN_REMOTE_ARGS ?=


How is this populated?

These are optional environment variables, Loki's RP-in-containers efforts made use of them so I included them here for completeness

…isn't supported by the Azure Ansible module at this time

s-fairchild · 2024-10-31T19:13:05Z

/retest

ventifus · 2024-11-18T18:28:05Z

Added code in the latest commit for https://issues.redhat.com/browse/ARO-12055, enabling MIWI support and local RP support via rp_mode=development

shubhadapaithankar

LGTM

openshift-ci · 2024-12-12T21:18:36Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: s-fairchild, shubhadapaithankar, tsatam, ventifus
Once this PR has been reviewed and has the lgtm label, please ask for approval from sudobrendan. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2024-12-21T01:51:46Z

New changes are detected. LGTM label has been removed.

openshift-ci · 2024-12-21T02:11:07Z

@ventifus: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e	`ad10a6a`	link	false	`/test e2e`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot requested review from anshulvermapatel and jaitaiwan September 27, 2024 23:48

ventifus changed the title ~~Initial Ansible commit.~~ Ansible cluster installation Sep 27, 2024

ventifus force-pushed the ansible-cluster-automation branch 3 times, most recently from 68d8215 to 8673260 Compare September 28, 2024 00:10

Initial Ansible commit.

56f7782

ventifus force-pushed the ansible-cluster-automation branch from 8673260 to 56f7782 Compare September 30, 2024 15:55

openshift-ci bot assigned lranjbar Oct 3, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 3, 2024

tsatam approved these changes Oct 3, 2024

View reviewed changes

openshift-ci bot assigned tsatam Oct 3, 2024

openshift-ci bot assigned SudoBrendan Oct 3, 2024

lb-ip-count is now generally available.

dfa69ac

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 7, 2024

Add candidate-4.17

6402b2c

ventifus force-pushed the ansible-cluster-automation branch from 6da33d3 to 6402b2c Compare October 7, 2024 23:18

ventifus added 2 commits October 9, 2024 10:06

Fix disconnected upgrades

ed06b5e

Add documentation; reconcile documentation with collection variables

d2c6460

ventifus force-pushed the ansible-cluster-automation branch from 3b8b3be to cf4915f Compare October 24, 2024 16:29

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 30, 2024

ventifus force-pushed the ansible-cluster-automation branch from cf4915f to 8fbdedd Compare October 30, 2024 15:33

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 30, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 30, 2024

ventifus force-pushed the ansible-cluster-automation branch from 8fbdedd to c1328e7 Compare October 30, 2024 16:18

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 30, 2024

s-fairchild suggested changes Oct 30, 2024

View reviewed changes

openshift-ci bot assigned s-fairchild Oct 30, 2024

Fall back to az network vnet create when requesting encryption, this …

83aa5f6

…isn't supported by the Azure Ansible module at this time

ventifus force-pushed the ansible-cluster-automation branch from c1328e7 to 83aa5f6 Compare October 30, 2024 16:42

Clean up old commented-out code

4af9d52

ventifus force-pushed the ansible-cluster-automation branch from fd7c762 to 4af9d52 Compare October 30, 2024 17:29

s-fairchild approved these changes Oct 31, 2024

View reviewed changes

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 31, 2024

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Nov 13, 2024

ventifus force-pushed the ansible-cluster-automation branch 5 times, most recently from 58273c0 to 7880c72 Compare November 14, 2024 21:27

Initial MIWI support

ae9af46

ventifus force-pushed the ansible-cluster-automation branch from 7880c72 to ae9af46 Compare November 16, 2024 00:51

Add cluster wide proxy support

fc1d69b

shubhadapaithankar approved these changes Dec 12, 2024

View reviewed changes

openshift-ci bot assigned shubhadapaithankar Dec 12, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 12, 2024

Add support for bring-your-own-NSG

ad10a6a

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Dec 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ansible cluster installation #3

Ansible cluster installation #3

ventifus commented Sep 27, 2024 •

edited

Loading

lranjbar commented Oct 3, 2024 •

edited

Loading

ventifus commented Oct 3, 2024

lranjbar commented Oct 30, 2024

lranjbar commented Oct 30, 2024

s-fairchild left a comment

s-fairchild Oct 30, 2024

ventifus Oct 30, 2024

s-fairchild Oct 30, 2024

ventifus Oct 30, 2024

s-fairchild Oct 30, 2024

ventifus Oct 30, 2024

s-fairchild Oct 30, 2024

ventifus Oct 30, 2024

s-fairchild Oct 30, 2024

ventifus Oct 30, 2024

s-fairchild Oct 30, 2024

ventifus Oct 30, 2024

s-fairchild Oct 30, 2024

ventifus Oct 30, 2024

s-fairchild Oct 30, 2024

ventifus Oct 30, 2024

s-fairchild Oct 30, 2024

ventifus Oct 30, 2024

s-fairchild Oct 30, 2024

ventifus Oct 30, 2024

s-fairchild commented Oct 31, 2024

ventifus commented Nov 18, 2024

shubhadapaithankar left a comment

openshift-ci bot commented Dec 12, 2024

openshift-ci bot commented Dec 21, 2024

openshift-ci bot commented Dec 21, 2024

		# private_link_service_network_policies:
		# "{% if outbound_type is defined and outbound_type == 'UserDefinedRouting' %}Disabled{% else %}{{ omit }}{% endif %}"

Ansible cluster installation #3

Are you sure you want to change the base?

Ansible cluster installation #3

Conversation

ventifus commented Sep 27, 2024 • edited Loading

What this PR does / why we need it:

lranjbar commented Oct 3, 2024 • edited Loading

ventifus commented Oct 3, 2024

lranjbar commented Oct 30, 2024

lranjbar commented Oct 30, 2024

s-fairchild left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

s-fairchild commented Oct 31, 2024

ventifus commented Nov 18, 2024

shubhadapaithankar left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Dec 12, 2024

openshift-ci bot commented Dec 21, 2024

openshift-ci bot commented Dec 21, 2024

ventifus commented Sep 27, 2024 •

edited

Loading

lranjbar commented Oct 3, 2024 •

edited

Loading