Create Customized POD Configs for PPDM Pod´s

Disclaimer: use this at your own Risk

why that ?

The PowerProtect DataManager Inventory Source uses Standardized Configurations and Configmaps for the Pods deployed by PPDM, e.gh. cProxy, PowerProtect Controller, as well as Velero.

In this Post, i will use an example to deploy the cProxies to dedicated nodes using Node Affinity. This makes Perfect sense if you want to separate Backup from Production Nodes.

More Examples could include using cni plugins like multus, dns configurations etc.

The Method described here is available from PPDM 19.10 and will be surfaced to the UI in Future Versions

1. what we need

The below examples must be run from a bash shell. We will use jq to modify json Documents.

2. Adding labels to worker nodes

in order to use Node Affinity for our Pods, we first need to label nodes for dedicated usage.

In this Example we label a node with tier=backup

A corresponding Pod Configuration example would look like:

apiVersion: apps/v1
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: tier
            operator: In # tag must be *in* tags
            values:
            - backup            
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent

We will use a customized template section later for our cProxy

So first, tag the Node(s) you want to use for Backup:

kubectl label nodes ocpcluster1-ntsgq-worker-local-2xl2z tier=backup

2. Create a Configuration Patch for the cProxy

We create manifest Patch for the cProxy from a yaml Document. This will be base64 encoded and presented as a Value to POD_CONFIG type to our Inventory Source The API Reference describes the Format of the Configuration.

The CPROXY_CONFIG variable below will contain the base64 Document

CPROXY_CONFIG=$(base64 -w0 <<EOF
---
metadata:
  labels:
    app: cproxy
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: tier
            operator: In
            values:
            - backup      
EOF
)    

3. Patching the inventory Source using the PowerProtect Datamanager API

You might want to review the PowerProtect Datamanager API Documentation

for the Following Commands, we will leverage some BASH Variables to specify your environment:

PPDM_SERVER=<your ppdm fqdn>
PPDM_USERNAME=<your ppdm username>
PPDM_PASSWORD=<your ppdm password>
K8S_ADDRESS=<your k8s api address the cluster is registered with, see UI , Asset Sources -->, Kubernetes--> Address>

3.1 Logging in to the API and retrieve the Bearer Token

The below code will read the Bearer Token into the TOKEN variable

TOKEN=$(curl -k --request POST \
  --url https://${PPDM_SERVER}:8443/api/v2/login \
  --header 'content-type: application/json' \
  --data '{"username":"'${PPDM_USERNAME}'","password":"'${PPDM_PASSWORD}'"}' | jq -r .access_token)

3.2 Select the Inventory Source ID based on the Asset Source Address

Select inventory ID matching your Asset Source Address :

K8S_INVENTORY_ID=$(curl -k --request GET https://${PPDM_SERVER}:8443/api/v2/inventory-sources \
--header "Content-Type: application/json" \
--header "Authorization: Bearer ${TOKEN}" \
| jq --arg k8saddress "${K8S_ADDRESS}" '[.content[] | select(.address==$k8saddress)]| .[].id' -r)

3.3 Read Inventory Source into Variable

With the K8S_INVENTORY_ID from above, we read the Inventory Source JSON Document into a Variable

INVENTORY_SOURCE=$(curl -k --request GET https://${PPDM_SERVER}:8443/api/v2/inventory-sources/$K8S_INVENTORY_ID \
--header "Content-Type: application/json" \
--header "Authorization: Bearer ${TOKEN}")

3.4 Adding the Patched cProxy Config to the Variable

Using jq, we will modify the Inventory Source JSON Document to include our base64 cproxy Config.

for that, we will add an list with content

"configurations": [
        {
          "type": "POD_CONFIG",
          "key": "CPROXY",
          "value": "someBase64Document"
        }
      ]

Other POD_CONFIGS key´s we could modify are POWERPROTECT_CONTROLLER and VELERO the below json code will take care for this:

INVENTORY_SOURCE=$(echo $INVENTORY_SOURCE| \
 jq --arg cproxyConfig "${CPROXY_CONFIG}" '.details.k8s.configurations += [{"type": "POD_CONFIG","key": "CPROXY", "value": $cproxyConfig}]')

3.4 Patching the Inventory Source in PPDM

We now use a POST request to upload the Patched inventory Source Document to PPDM

curl -k -X PUT https://${PPDM_SERVER}:8443/api/v2/inventory-sources/$K8S_INVENTORY_ID \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $TOKEN" \
-d "$INVENTORY_SOURCE"

We can verify the patch by checking the PowerProtect Configmap in the PPDM Namespace.

kubectl get configmap  ppdm-controller-config -n powerprotect -o=yaml

The configmap must now contain cproxy-pod-custom-config in data

The next Backup job will now use the tagged node(s) for Backup !

3.5 Start a Backup and verify the node affinity for cproxy pod

First, identify the node you labeled

kubectl get nodes -l 'tier in (backup)'

once the backup job creates the cproxy, this should be done on one of the identified nodes:

kubectl get pods -n powerprotect -o wide

This concludes the Patching of cproxy Configuration, stay tuned for more

Installation and troubleshooting Tips for PPDM 19.11 in disconnected OpenShift 4.10 Environments

Disclaimer: use this at your own Risk

why that ?

I am happily running some disconnected OpenShift Clusters on VMware and AzureStack Hub. This Post is about how to Protect a Greenfield Deployment of OpenShift 4.10 with PPDM 19.11, running on vSphere.

OpenShift Container Platform 4.10 installs the vSphere CSI Driver Operator and the vSphere CSI driver by default in the openshift-cluster-csi-drivers namespace.
This setup is different from a user Deployed CSI driver or CSI Drivers as a Process like in TKGi.

In this Post, i will point out some Changes required to make OpenShift 4.10 fresh installs work with PPDM 19.11

1. what we need

In order to support Disconnected OpenShift Clusters, we need to prepare our environment to Host certain Images and Operators on a local Registry.

1.1 Mirroring the Operator Catalog for the OADP Provider

PPDM utilized the RedHat OADP (OpenShift APIs for Data Protection) Operator vor Backup and Restore. Therefore, we need to Replicate redhat-operators from the index image registry.redhat.io/redhat/redhat-operator-index:v4.10 This is well documented on Using Operator Lifecycle Manager on restricted networks, specifically if you want to filter the Catalog to a specific list of Packages.

If you want to mirror the entire Catalog, you can utilize oc adm catalog mirror

Once the Replication is done, make sure apply the ImageContentSourcePolicy and the new CatalogSource itself.

1.2 Mirroring the DELL PowerProtect Datamanager and vSphere/Velero Components

The Following Images need to be Replicated to your local Registry:

2. Setup

2.1 Configure PPDM to point to the local Registry

On the PPDM Server, point the CNDM (Cloud Native Datamover )Service to the Registry that is hosting your PPDM and Velero Images by editing /usr/local/brs/lib/cndm/config/application.properties Put the FQDN for k8s.docker.registry to reflect your local registry:

k8s.docker.registry=harbor.pks.home.labbuildr.com

This will instruct cndm to point to your registry when creating configmaps for PPDM You need to restart the Service in order for changes to affect.

cndm restart

2.2 Configure a Storage Class with volumeBindingMode: Immediate

The vSphere CSI Operator Driver storage class uses vSphere’s storage policy. OpenShift Container Platform automatically creates a storage policy that targets datastore configured in cloud configuration. However, this uses volumeBindingMode: WaitForFirstConsumer. In order to restore PVC´s to a new Namespace, we need a StorageClass with volumeBindingMode: Immediate Make sure the StoragePolicyName reflects ure existing Policy ( from Default Storage Class )

oc apply -f - <<EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: thin-csi-immediate
  annotations:
    storageclass.kubernetes.io/is-default-class: 'true'
provisioner: csi.vsphere.vmware.com
parameters:
  StoragePolicyName: openshift-storage-policy-ocs1-nv6xw # <lookup you policy>
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate
EOF      

2.3 Deploy the RBAC Templates from PPDM

RBAC Templates containing PPDM CRD´s, RoleBindings and ClusterRoleBindings,Roles and ConfigMaps and ServiceAccount Definitions can found on PPDM Server in /usr/local/brs/lib/cndm/misc/rbac.tar.gz.
Always use the Latest version that comes with your PPDM version !
Once you copied and extracted the Templates, apply the to your cluster:

oc apply -f ../ocs_vsphere/ppdm-rbac/rbac/ppdm-discovery.yaml
oc apply -f ../ocs_vsphere/ppdm-rbac/rbac/ppdm-controller-rbac.yaml

2.4 Retrieve the PPDM Discovery Service Account Token

Applying the above Templates has also created the Powerprotect namespace and Discovery Token. Retrieve the token with

export PPDM_K8S_TOKEN=$(oc get secret \
"$(oc -n powerprotect get secret \
 | grep ppdm-discovery-serviceaccount-token \
 | head -n1 | awk '{print $1}')" \
-n powerprotect --template={{.data.token}} | base64 -d      

2.5 Add the OpenShift Cluster to PPDM

If you not have already enabled the Kubernetes Asset Source in PPDM, do so now. To do so, go to Infrastructure > AssetSources > + In the New Asset Source Tab, Click on Enable Source for the Kubernetes Asset Source

Select the Kubernetes tab and click Add. Provide your Cluster information and click on Add credentials. Provide a Name for the Discovery Service Account and Provide the Token extracted in the Previous Step

Click on Verify to Validate the Certificate of the OpenShift Cluster hint: if you use unknown root ca, follow the user guide to add the ca to ppdm using the ppdmtool

The the powerprotect controller and Configmap should now being deployed to powerprotect namespace

To view the configmap, execute

oc get configmap  ppdm-controller-config -n powerprotect -o=yaml

to view the logs, execute:

oc logs "$(oc -n powerprotect get pods \
| grep powerprotect-controller \
| awk '{print $1}')" -n powerprotect -f

if the controller is waiting for velero-ppdm ip,

then probably your oadp-operator subscription is pointing to the wrong operator catalog source. View your subscription with

oc describe subscription redhat-oadp-operator -n velero-ppdm

  Conditions:
    Last Transition Time:  2022-04-22T08:54:35Z
    Message:               targeted catalogsource openshift-marketplace/community-operators missing
    Reason:                UnhealthyCatalogSourceFound
    Status:                True
    Type:                  CatalogSourcesUnhealthy
    Message:               constraints not satisfiable: no operators found from catalog community-operators in namespace openshift-marketplace referenced by subscription oadp-operator, subscription oadp-operator exists
    Reason:                ConstraintsNotSatisfiable
    Status:                True
    Type:                  ResolutionFailed
  Last Updated:            2022-04-22T08:54:35Z
Events:                    <none>

In this case, we need to patch the Subscription to reflect the Catalog Name we created for our custom Operator Catalog ( in this example, redhat-operator-index):

oc patch subscription redhat-oadp-operator -n velero-ppdm --type=merge -p '{"spec":{"source": "redhat-operator-index"}}'

This should deploy the RedHat OADP Operator in the velero-ppdm namespace. You can view the Operator Status from your OpenShift Console

Now it is time to watch the Pods Created in the velero-ppdm Namespace:

oc get pods -n velero-ppdm
      

The Backup driver fails to deploy, so we need to have a look at the logs:

oc logs "$(oc -n velero-ppdm get pods | grep backup-driver | awk '{print $1}')" -n velero-ppdm -f    

Obviously, the OpenShift Cluster Operator deployed CSI drivers, but no configuration was made for the velero-vsphere-plugin to point to a default secrets for velero backup-driver (different from standard CSI Installation ) So we need to create a secret from a vSphere config File hosting the clusterID and the vSphere Credentials, and apply a configmap to ppdm to use the secret.

Use the following example to create a config file with your vCenter csi account:

cat > csi-vsphere.conf <<EOF
[Global]
cluster-id = "$(oc get clusterversion -o jsonpath='{.items[].spec.clusterID}{"\n"}')"
[VirtualCenter "vcsa1.home.labbuildr.com"]
insecure-flag = "true"
user = <your vsphere csi user>
password = <your csi acccount secret>
port = 443
datacenters = "<your dc>"
EOF   

Create a Secret bfrom that file using:

oc -n velero-ppdm create secret generic velero-vsphere-config-secret --from-file=csi-vsphere.conf  

Apply the configmap for the velero vSphere Plugin

oc -n velero-ppdm apply -f - <<EOF
kind: ConfigMap
apiVersion: v1
metadata:
  name: velero-vsphere-plugin-config
  namespace: velero-ppdm
  labels:
    app.kubernetes.io/part-of: powerprotect.dell.com
data:
  cluster_flavor: VANILLA
  vsphere_secret_name: velero-vsphere-config-secret
  vsphere_secret_namespace: velero-ppdm
EOF

And delete the existing backup-driver pod

oc delete pod "$(oc -n velero-ppdm get pods | grep backup-driver | awk '{print $1}')" -n velero-ppdm

We can see the results with:

oc logs "$(oc -n velero-ppdm get pods | grep backup-driver | awk '{print $1}')" -n velero-ppdm

3. Run a backup !

Follow the Admin Guide to create a Protection Policy for your Discovered OpenShift Assets and select a Namespace with a PVC. Run the Policy and examine the Status Details.

Unfortunately, PPDM tries to trigger a CSI based Backup instead of a FCD based PVC Snapshot. This is because upon initial discovery, PPDM did not correctly identify a “VANILLA_VSPHERE” Cluster, as the cluster-csi-operator uses a different install location for the csi drivers. This only Occurs on 4.10 Fresh installs, not on updates from 4.9 to 4.10 !

What we need to do now, is patching the Inventory Source of Our OpenShift Cluster in PPDM using the REST API.

4. Patching a PPDM Inventory Source for fresh deployed 4.10 Clusters

4.1 Get and use the bearer token for subsequent requests

      
PPDM_SERVER=<your ppdm fqdn>
PPDM_USERNAME=<your ppdm username>
PPDM_PASSWORD=<your ppdm password>
TOKEN=$(curl -k --request POST \
  --url "https://${PPDM_SERVER}:8443/api/v2/login" \
  --header 'content-type: application/json' \
  --data '{"username":"'${PPDM_USERNAME}'","password":"'${PPDM_PASSWORD}'"}' | jq -r .access_token)
      

4.2 Get the inventory id´s for vCenter and the OpenShift cluster

      
K8S_INVENTORY_ID=$(curl -k --request GET "https://${PPDM_SERVER}:8443/api/v2/inventory-sources" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer ${TOKEN}" \
| jq '[.content[] | select(.address=="api.ocs1.home.labbuildr.com")]| .[].id' -r)

VCENTER_ID=$(curl -k --request GET "https://${PPDM_SERVER}:8443/api/v2/inventory-sources" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer ${TOKEN}" \
| jq '[.content[] | select(.address=="vcsa1.home.labbuildr.com")]| .[].id' -r)
      

4.3 query the distribution type field of the kubernetes inventory source by using above K8S_INVENTORY_ID

 
curl -k --request GET "https://${PPDM_SERVER}:8443/api/v2/inventory-sources/$K8S_INVENTORY_ID" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer ${TOKEN}" | jq .details.k8s.distributionType
 

If the Distribution type shows NON_VSPHERE, we need to patch it

4.4 Read the inventory into variable:

 
INVENTORY_SOURCE=$(curl -k --request GET "https://${PPDM_SERVER}:8443/api/v2/inventory-sources/$K8S_INVENTORY_ID" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer ${TOKEN}")
 

4.5 Patch the variable content using jq:

 
INVENTORY_SOURCE=$(echo $INVENTORY_SOURCE \
| jq --arg distributionType "VANILLA_ON_VSPHERE" '.details.k8s.distributionType |= $distributionType')
INVENTORY_SOURCE=$(echo $INVENTORY_SOURCE \
| jq --arg vCenterId "$VCENTER_ID" '.details.k8s.vCenterId |= $vCenterId')
 

4.6 validate the variable using jq:

 
echo $INVENTORY_SOURCE| jq .details.k8s
 

4.7 And Patch that PPDM !!!

curl -k -X PUT "https://${PPDM_SERVER}:8443/api/v2/inventory-sources/$K8S_INVENTORY_ID" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $TOKEN" \
-d "$INVENTORY_SOURCE"

5. Run the Protection Policy Again

In the Job List in PPDM, select the failed job and click on restart

The Job should now run Successful, and take PVC Snapshots (using First Class Disks) instead of CSI Snapshots

Update Dell PowerProtect Datamanager Appliance using Ansible and REST API

Disclaimer: use this at your own Risk

why that ?

Automation is everywhere. Automation should your standards. Automation should be based on (Rest) API´s.
As our customers broadly use Ansible to automate standard IT Tasks, i created this Example use case to manage Dell PowerProtect Datamanager´s Update lifecycle.
My Friend Preston De Guise has written a nice Blogpost on PowerProtect Data Manager 19.11 – What’s New, and Updating

We will do all of his outlined steps to update a PowerProtect Datamanager from 19.10 to 19.11 using Ansible.

The individual API Calls are Organized in Ansible roles and will be executed from corresponding Playbooks

what we need

In general, all of the following playbooks / roles are built with the Ansible URI module ( with one exception :-) ) They have been tested from Ubuntu Linux ( running from WSL2 ), best “Ubuntu 20.04.4 LTS” for python 3.8 support

where to get the REST API calls

we can find the REST API calls required / used in my Examples on DELL Technologies Developer Portal

i am happy to share my ansible roles per request for now. so let us walk trough a typical update scenario

1. Uploading the Update Package

1.1 the get_ppdm_token role to authenticate with the api endpoint

All of the playbooks require to authenticate via the API endpoint and use a Bearer token for Subsequent requests. so i use a role called get_ppdm_token to retrieve the token from the API:

# role to get a Bearer Token from the PPDM API
- name: Get PPDM Token
  uri:
    method: POST
    url: ":/login"
    body:
      username: ""
      password: ""
    body_format: json 
    status_code: 200
    validate_certs: false
  register: result  
  when: not ansible_check_mode 
- set_fact:
    access_token: ""

1.2 the upload_update role for uploading a package to PPDM

to upload an update package, we have to use a curl via an ansible shell call ( this being the one exception :-) ), as an upload vi URI Module fails for files > 2GB

# note: using curl here as uploads >=2GB still fail in ansible uri module ....
- name: Uploading Update File , this may take a while
  shell: 'curl -ks :/upgrade-packages \
      -XPOST \
      -H "content-type: multipart/form-data"  \
      -H "authorization: Bearer " \
      --form file=@'

1.3 Playbook to upload the Update Package

The following snippet is an example of an Upload Playbook

# Example Playbook to configure PPDM Certificates
- name: Update PPDM
  hosts: localhost
  connection: local
  vars_files: 
    - ./vars/main.yml
  tasks:
  - name: Setting Base URL
    set_fact: 
      ppdm_baseurl: "https://{{ ppdm_fqdn | regex_replace('^https://') }}"     
  - name: Get PPDM Token for https://{{ ppdm_fqdn | regex_replace('^https://') }}
    include_role: 
      name: get_ppdm_token
    vars: 
      ppdm_password: "{{ ppdm_new_password }}"
  - debug: 
      msg: "{{ access_token }}"
      verbosity: 1
    name: do we have a token ?  
  - name: find PowerProtect update software at input mapping
    find:
      paths: "{{ lookup('env','PWD') }}/files"
      patterns: 'dellemc-ppdm-upgrade-sw-*.pkg'
    register: result
  - name: Setting Upload Filename
    set_fact:
      upgrade_file: "{{ result.files[0].path }}"
    when: result.files is defined     
  - name: parsed upload file
    debug: 
      msg: "{{ upgrade_file }}"
  - name: Uploading Update
    include_role: 
      name: upload_update
    vars: 
      upload_file: "{{ upgrade_file }}"   
      

The Playbook will run like this:

2. Pre Checking the Update

The Update Pre Check will inform you about Required Actions need to be taken in order to make the update succeed. It will also show us Warnings for issues we might correct prior the Update.

2.1 the get_ppdm_update_package role for getting the Update ID

PPDM has an API endpoint to get all Updates in the System. This could be active or historical Updates. The Response Contains JSON Documents with specific information and state about the Update

- name: Define uri with package ID
  set_fact: 
    uri: "{{ ppdm_baseurl }}:{{ ppdm_port }}{{ ppdm_api_ver }}/upgrade-packages/{{ id }}"
  when: id  | default('', true) | trim != ''
- name: Define uri without package id
  set_fact: 
    uri: "{{ ppdm_baseurl }}:{{ ppdm_port }}{{ ppdm_api_ver }}/upgrade-packages"
  when: id  | default('', false) | trim == ''
- name: Rewrite uri with filter
  set_fact: 
    uri: "{{ uri }}?filter={{ filter | urlencode }}"  
  when: filter  | default('', true) | trim != ''
- name: Getting Update Package State
  uri:
    method: GET
    url: "{{ uri }}"
    body_format: json
    headers:
      accept: application/json
      authorization: "Bearer {{ access_token }}"    
    status_code: 200,202,403
    validate_certs: false
  register: result  
  when: not ansible_check_mode 
- set_fact:
    update_package: "{{ result.json.content }}"
  when: id  | default('', false) | trim == ''
- set_fact:
    update_package: "{{ result.json }}"
  when: id  | default('', true) | trim != ''      
- debug:
    msg: "{{ update_package }}"
    verbosity: 1
      

This task allows us to get info a specific update by using the update ID, or filter an update by type ( e.g. ACTIVE or HISTORY )

2.2 example Playbook to get an ACTIVE Update Content

# Example Playbook to get an ACTIVE Package
#
# Note: Api is still called Upgrade, but Upgrade is for Hardware only !
# SO tasks will be named update but execute Upgrade
#
- name: get PPDM Update Package
  hosts: localhost
  connection: local
  vars_files: 
    - ./vars/main.yml
  tasks:
  - name: Setting Base URL
    set_fact: 
      ppdm_baseurl: "https://{{ ppdm_fqdn | regex_replace('^https://') }}"     
  - name: Get PPDM Token for https://{{ ppdm_fqdn | regex_replace('^https://') }}
    include_role: 
      name: get_ppdm_token
    vars: 
      ppdm_password: "{{ ppdm_new_password }}"
  - debug: 
      msg: "{{ access_token }}"
      verbosity: 1
    name: do we have a token ?  
  - name: get Update Package
    include_role: 
      name: get_ppdm_update_package
    vars:
      filter: 'category eq "ACTIVE"' 
  - debug: 
      msg: "{{ update_package }}"

      

This example shows the JSON response for the Update Package with Essential information for example Versions,Required Passwords/Reboot, Updated EULA´s, ID etc:

2.3 the precheck_ppdm_update_package role

After reviewing the Information, we will trigger the Pre Check using the precheck endpoint. This will transform the Package state to Processing. The below Precheck role will wait for the state AVAILABLE and the Validation Results:

# Example Playbook to Precheck PPDM Update
#
# Note: Api is still called Upgrade, but Upgrade is for Hardware only !
# SO tasks will be named update but execute Upgrade
#
- name: precheck PPDM Update Package
  uri:
    method: POST
    url: "{{ ppdm_baseurl }}:{{ ppdm_port }}{{ ppdm_api_ver }}/upgrade-packages/{{ id }}/precheck"
    body_format: json
    headers:
      accept: application/json
      authorization: "Bearer {{ access_token }}"    
    status_code: 202,204,401,403,404,409,500
    validate_certs: false
  register: result  
  retries: 10
  delay: 30
  until: result.json.state is defined and result.json.state == "PROCESSING"
  when: not ansible_check_mode  
- set_fact:
    update_precheck: "{{ result.json }}"
- debug:
    msg: "{{ update_precheck }}"
    verbosity: 1
- name: Wait  Update Package State
  uri:
    method: GET
    url: "{{ ppdm_baseurl }}:{{ ppdm_port }}{{ ppdm_api_ver }}/upgrade-packages/{{ id }}"
    body_format: json
    headers:
      accept: application/json
      authorization: "Bearer {{ access_token }}"    
    status_code: 200,202,403
    validate_certs: false
  register: result 
  retries: 10 
  delay:  10
  when: not ansible_check_mode 
  until: result.json.validationDetails is defined and result.json.state == "AVAILABLE"
- set_fact:
    validation_details: "{{ result.json.validationDetails }}"   
- debug:
    msg: "{{ validation_details }}"
    verbosity: 1    
      

2.4 Running the Precheck Playbook

The Corresponding Playbook for above task will may look like this:

# Example Playbook to Precheck PPDM Update
#
# Note: Api is still called Upgrade, but Upgrade is for Hardware only !
# SO tasks will be named update but execute Upgrade
#
- name: Upgrade PPDM
  hosts: localhost
  connection: local
  vars_files: 
    - ./vars/main.yml
  tasks:
  - name: Setting Base URL
    set_fact: 
      ppdm_baseurl: "https://{{ ppdm_fqdn | regex_replace('^https://') }}"     
  - name: Get PPDM Token for https://{{ ppdm_fqdn | regex_replace('^https://') }}
    include_role: 
      name: get_ppdm_token
    vars: 
      ppdm_password: "{{ ppdm_new_password }}"
  - debug: 
      msg: "{{ access_token }}"
      verbosity: 1
    name: do we have a token ?  
  - name: get Update Package
    include_role: 
      name: get_ppdm_update_package
    vars:
      filter: 'category eq "ACTIVE"' 

  - name: delete_ppdm_upgrade
    when: update_package[0] is defined and update_package[0].upgradeError is defined
    include_role: 
      name: delete_ppdm_update_package
    vars:
      id: "{{ update_package[0].id }}"
  - name: precheck_ppdm_upgrade
    when: update_package[0] is defined and update_package[0].upgradeError is not defined
    include_role: 
      name: precheck_ppdm_update_package
    vars:
      id: "{{ update_package[0].id }}"
  - debug:
      msg: "{{ validation_details }}"
      verbosity: 0          
      

The Playbook will output the validation Details:

3. Executing the Upgrade

Once the Update Validation has passed, we can execute the Update

3.1 the install_ppdm_update_package role

The below example task shows the upgrade-packages endpoint to be called A JSON Body needs to be provided with additional information. We wil create the Body from the Calling Playbook

- name: install PPDM Update Package
  uri:
    method: PUT
    url: "{{ ppdm_baseurl }}:{{ ppdm_port }}{{ ppdm_api_ver }}/upgrade-packages/{{ id }}?forceUpgrade=true"
    body_format: json
    body: "{{ body }}"
    headers:
      accept: application/json
      authorization: "Bearer {{ access_token }}"    
    status_code: 202,204,401,403,404,409,500
    validate_certs: false
  register: result  
  when: not ansible_check_mode  
- set_fact:
    update: "{{ result.json }}"
- debug:
    msg: "{{ update }}"
    verbosity: 0
      

3.2 the check_update_done role

Once the Update is started, PPDM sysmgr and other components will shut down. PPDM has a specific API Port to Monitor the Update( also from the UI, Port 14443) The check_update_done role will wait until the Update reaches 100%

- name: Wait for Appliance Update Done, this may take a few Minutes
  uri:
    method: GET
    url: "{{ ppdm_baseurl }}:14443/upgrade/status"
    status_code: 200 
    validate_certs: false
  register: result  
  when: not ansible_check_mode 
  retries: 100
  delay: 30
  until: ( result.json is defined and result.json[0].percentageCompleted == "100" ) or ( result.json is defined and result.json[0].upgradeStatus =="FAILED" )
- debug: 
    msg: "{{ result.json }}"
    verbosity: 0
  when: result.json is defined  
      

3.3 Playbook to install the Update

to install the update, we will modify the update-package JSON document to change the state field to INSTALLED. Also, we will approve updated EULA Settings and trust the Package Certificate. The changes will be merged from the Playbook.

state: INSTALLED
certificateTrustedByUser: true
eula: 
  productEulaAccepted: true # honor EULA Changes
  telemetryEulaAccepted: true  # honor EULA Changes
      

The corresponding Playbook looks like

# Example Playbook to Install PPDM Update
#
# Note: Api is still called Upgrade, but Upgrade is for Hardware only !
# SO tasks will be named update but execute Upgrade
#
- name: Upgrade PPDM
  hosts: localhost
  connection: local
  vars_files: 
    - ./vars/main.yml
  tasks:
  - name: Setting Base URL
    set_fact: 
      ppdm_baseurl: "https://{{ ppdm_fqdn | regex_replace('^https://') }}"     
  - name: Get PPDM Token for https://{{ ppdm_fqdn | regex_replace('^https://') }}
    include_role: 
      name: get_ppdm_token
    vars: 
      ppdm_password: "{{ ppdm_new_password }}"
  - debug: 
      msg: "{{ access_token }}"
      verbosity: 1
    name: do we have a token ?  
  - name: get Update Package
    include_role: 
      name: get_ppdm_update_package
    vars:
      filter: 'category eq "ACTIVE"' 

  - name: get_ppdm_upgrade_by ID
    when: update_package[0] is defined and update_package[0].upgradeError is not defined
    include_role: 
      name: get_ppdm_update_package
    vars:
      id: "{{ update_package[0].id }}"

  - name: merge update body
    vars:
      my_new_config:
        state: INSTALLED
        certificateTrustedByUser: true
        eula: 
          productEulaAccepted: true # honor EULA Changes
          telemetryEulaAccepted: true  # honor EULA Changes
    set_fact:
      update_package: "{{ update_package | default([]) | combine(my_new_config, recursive=True) }}"
  - debug: 
      msg: "{{ update_package }}"
  - name: install_ppdm_upgrade
    when: update_package is defined and update_package.state == "INSTALLED"
    include_role: 
      name: install_ppdm_update_package
    vars:
      id: "{{ update_package.id }}"
      body: "{{ update_package }}"
  - name: Validate Update Started
    when: update_package is defined and update_package.upgradeError is not defined
    include_role: 
      name: get_ppdm_update_package
    vars:
      id: "{{ update_package.id }}"   
  - name: Check Update State  https://{{ ppdm_fqdn | regex_replace('^https://') }}
    include_role:
      name: check_update_done 
      

We can follow the Upgrade Process from the Ansible Playbook:

And also use the Webbrowser at “https://:14443" to see the Update Status:

basic install for test and demo or just in case

Disclaimer: use this at your own Risk

why that ?

Due to an Azure Marketplace API Change we are currently investigating, some combined Marketplace Applications in Azure no longer deploy. This also affect´s DellEMC´s Combined Deployments for Avamar w./DataDomain as well as PowerProtect w./ DataDomain.
This Example will deploy a Fresh PowerProtect Standalone Appliance into a Fresh Resource group for basic testing.
This guide is to give you an understanding on how to accept marketplace terms and deploy anb appliance from a marketplace image using the cli.
I will create a more comprehensive guide for Custom Installs soon.

The Problem lok like this

Uuups, an API Changed. So the blade does not give us a better answer then opening a case.

so with that, we would not be able to deploy the appliance from the Marketplace.

However, we could still use terraform, ARM Templates or … az cli.

So for quick and dirty, I just guide you through an az-cli process:

identifying the template

first of all, we need to get the current image offer for dellemc powerprotect

az vm image list --all --offer ppdm --publisher dellemc

This will show you all required information to deploy the image. But before we start, we need to accept the Marketplace Image Terms for that image.
For that, we just use az vm image terms accept with the urn:

az vm image terms accept --urn dellemc:ppdm_0_0_1:powerprotect-data-manager-19-7-0-7:19.7.0

We also might want to look on what the image includes as image resources:

az vm image show --urn dellemc:ppdm_0_0_1:powerprotect-data-manager-19-7-0-7:19.7.0

Deploying the Image

first of all we need to create a resource group for ppdm, if not deploying to an existing one. in my example, I am going to deploy into a resource group ppdm_from_cli in location germanywestcentral

if you do not ave already created a resource group for you deployment, do so with

export RESOURCE_GROUP="ppdm_from_cli"
export LOCATION="germanywestcentral"
az group create --resource-group ${RESOURCE_GROUP} \
--location ${LOCATION}

next, we will cerate the VM form the Marketplace image using az vm create. az vm create will create all required resources ( vnet, NIC, NSG, pulicIp ) unless we specify a specific configuration.
Just for the test, we will accept the standard Parameters. In Real World Scenarios, people would deploy to an existing Network that might be managed by CloudAdmin´s.
You will find a more comprehensive Guide on that here soon.
if you do not want a Public IP at this point, add –public-ip-address “”

for now, we just do:

export VM_NAME="ppdm1"
az vm create --resource-group ${RESOURCE_GROUP} \
 --name ${VM_NAME} \
 --image dellemc:ppdm_0_0_1:powerprotect-data-manager-19-7-0-7:19.7.0 \
 --plan-name powerprotect-data-manager-19-7-0-7 \
 --plan-product ppdm_0_0_1 \
 --plan-publisher dellemc \
 --size Standard_D8s_v3

Note: the required for PPDM is a Standard_D8s_v3 ! do not change that !

in order to access the vm via Public IP for Configuration, we need to open 443 on the NSG. The default NSG is “${VM_NAME}NSG”

az network nsg rule create --name https \
--nsg-name "${VM_NAME}NSG" \
--resource-group ${RESOURCE_GROUP} \
--protocol Tcp --priority 300 \
--destination-port-range '443'

Give the System a few moments to but up and configure basic things
Meanwhile, you might want to look at you deployed resources from the Portal:

Try to connect to the appliance on https://[public_ip] after some minutes. it should bring you to the Appliance Fresh Install page

You can now proceed to configure the appliance. For this, follow the PowerProtect Data Manager 19.7 Azure Deployment Guide from our support site.

if you want to delete you deployment just use

az group delete --resource-group ${RESOURCE_GROUP}

keep my K8S up to date with Azure…

Disclaimer: use this at your own Risk

Recently I tried to deploy newer versions of Kubernetes using AKS Engine to my AzureStack hub, but i failed for some odd reasons. That made me thinking about how do get newer Versions deployed to my AzureStack Hub using aks-engine.

Why should you want a newer version at all? Well, the answer is fairly simple. With K8s dot releases essentially coming monthly, one want’s to stay current with Security fixes. Patching is not an option. I love the image based approach where i simply replace the OS Image. The longer a VM runs, the more vulnerable it get’s. Deleting a VM an rebuild with a new Image is by far the most convenient and secure approach. aks-egine does a pretty good job in upgrading to newer image versions…. if available.

The Problem

If you want to use your own Image, or try a newer AKS Engine release, the Deployment will try to download the missing Packages from https://kubernetesartifacts.azureedge.net/kubernetes/ and the according Docker Images from mcr.microsoft.com/oss/kubernetes/

This Process is most likely to fail for slow Networks, as the download scripts have timeouts of 60 seconds per package.

My First idea was to patch the deployment scripts. While looking at the Code on GitHub, i found it might be much more convenient to create updated aks base images and put them onto my AzureStack Marketplace.

The Current State

As of this Post, the “Official” Documented version of AKS-Engine for AzureStack Hub is v0.55.4.
This Version has Built In Support for Kubernetes up to 1.17.11

Preseeded deployments requires the Following Marketplace Images to be deployed:

	AzureStackAKSUbuntu1604OSImageConfig = AzureOSImageConfig{
		ImageOffer:     "aks",
		ImageSku:       "aks-engine-ubuntu-1604-202007",
		ImagePublisher: "microsoft-aks",
		ImageVersion:   "2020.09.14",
	}

The Images have the following K8S Version Pre-Seeded:

K8S_VERSIONS="
1.17.11-azs
1.17.9-azs
1.16.14-azs
1.16.13-azs
1.15.12-azs
1.15.11-azs
"

However, while reading the Release Notes for AKS-Engine on GitHub, i found newer versions exist with newer Image Support:

Yes, it says v0.60.0 would support up to K8S-AzS v1.18.15, but that is NOT in the current image.

This would require image 2021.01.28, as per chore: rev 2021.01.28

Building a new Image

While reading and Browsing the aks-engine GitHub releases, i found it would be a convenient way for me to create new microsoft-aks images using packer the way Microsoft does…
The Packer building Process used my microsoft is essentially an automated approach used in their Pipelines to create new Azure Images.
So why not just use this for AzureStackHub ( keep in mind, it’s like Azure )

As you can see from the Changelog, aks-engine has newer support for AzureStack Hub, however, aks-engine would complain about a missing aks sku version.

To create a newer image offer, we will use packer and the artifacts from the aks-engine GitHub repository.
But before we start, we will need to do some prerequisite.

In a Nutshell, the Process will deploy an Ubuntu Base VM in Azure along with a temporary resource group, pull all supported docker Containers for aks from mcr an pre-seed all software versions. Some hardening and updates will run, and once finished and cleaned up, a new image will be provided along with the SAS Url to download.

Requirements

I run the imaging process from an Ubuntu 18.04 Machine. you can use a VM, WSL2 or any Linux host / Docker Image.

Install the following Packages:

• Packer from https://www.packer.io/downloads
• make utils ( sudo apt install make )
• git ( sudo apt install git)
• an Azure account ( the images will be created in Public Azure)

We also would need an Azure Service Principal in an Azure Subscription, as the build process runs in Azure.

Creating a v0.60.0 Compliant image

First of all, use git to clone into the aks-engine repo

git clone git@github.com:Azure/aks-engine.git
cd aks-engine

every release of aks sitś in a new bracnch to get a list of all released versions of aks-engine, simply type

git branch -r| grep release

as we want to create a v0.60.0 compliant image, we checkout the corresponding release branch:

git checkout release-v0.60.0

for the packer process, microsoft automatically creates a settings.json file. to pass required parameters for the file, we export some shell variables for the init process:
(Replace ARM Parameters with your values)

export SUBSCRIPTION_ID=<ARM_SUBSCRIPTION_ID>
export TENANT_ID=<ARM_TENANT_ID>
export CLIENT_SECRET=<ARM_CLIENT_SECRET>
export CLIENT_ID=<ARM_CLIENT_ID>
export AZURE_RESOURCE_GROUP_NAME=packer ( an existing resource group ) 
export AZURE_VM_SIZE=Standard_F4
export AZURE_LOCATION=germanywestcentral
export UBUNTU_SKU=16.04

once those variables are set, we can run the initialization of the Environment.

first, we connect programmatic to the azure environment with make az-login

make  az-login

if the login is successful, you SP and Credentials worked.

We can now use the make init-packer command to initialize the Packer Environment (e.g. create the storage account)

make init-packer

*note: i skipped the az-login in the picture above as it shows credentials :-)

once done with the init process, we can start to build our image.

make build-packer

Sit back and relax , the Process will take
a good amount of time as a lot of stuff will be seeded into the image.

Once the Image has been created, you will be presented with URL’s to download the image / template.
We need the “OSDiskUriReadOnlySas” URL now, as we will use it to add the Image to the AzureStack.

Review your Image SKU from /pkg/api/azenvtypes.go .
For example, for an aks-engine-ubuntu-1804-202007 image:

	AKSUbuntu1804OSImageConfig = AzureOSImageConfig{
		ImageOffer:     "aks",
		ImageSku:       "aks-engine-ubuntu-1804-202007",
		ImagePublisher: "microsoft-aks",
		ImageVersion:   "2021.01.28",
	}

Uploading the Image to AzureStackHub

Goto you AzureStackHub AdminPortal. Click on Compute > VM Images > Add to add a new Image.
Fill in the Image Offer/SKU/Publisher/Vesrion for above, and paste the OSDiskUriReadOnlySas in “OS disk blob URi”.

It will take a while for the Image Upload to be finished. Once the Image state changed fro creating to succeeded, you are ready to test a new deployment with the aks-engine version you used to build the image.

Here are 2 Examples of AKS Nodes running different image sku’s in different Clusters:

For an 16.04, engine v0.55.4 deployed Cluster running K8S v1.17.11:

For an 18.04, engine v0.60.0 deployed Cluster running K8S v1.18.15:

I am running a lot of different AKS Version in my lab with now issues.
In the configs i am using, i run them with the azure cni and calico deployed.
Happy k8s-ing