A production kubernetes cluster

 TLDR

1. An HA nginx cluster of a minimum of  3 VMs with tls
2. An HA etcd cluster of a minimum of 3 VMs with tls
3. Install k8s cluster with at least 3 master nodes using the external etcd cluster 
4. nginx will be used to load balance amongst all 3 master nodes, All connections to k8s including the api calls will go through nginx 
5. Use the external ceph cluster for main workloads
6. Use the local rook-ceph cluster which uses the drives in the physical worker nodes.

Networks

1. Restricted network
2. Public network

Storage

nodes: 3

Install a 3 node ceph cluster.

Hypervisors

nodes: 3

Node 1 will contain the following vms

1. master1
2. nginx1
3. etcd1
4. worker1

Node 2 will contain the following vms

1. master2
2. nginx2
3. etcd2
4. worker2

Node 3 will contain the following vms

1. master3
2. nginx3
3. etcd3
4. worker3

K8s

K8s will be installed on 3 VMs + X physical nodes as workers. All these hypervisors, VMs and k8s will be installed in restricted network. Public network ip addresses will be used for metallb loadbalancer and apps on k8s will use these public network ip addresses. The nginx (not the nginx cluster of VMs) ingress controller deployed on the k8s via the helm chart will take care of ingress.