scripts: added sharing objectstorage for local cluster with ingress

[shafi-elastisys]

Warning

This is a public repository, ensure not to disclose:

• personal data beyond what is necessary for interacting with this pull request, nor
• business confidential information, such as customer names.

What kind of PR is this?

Required: Mark one of the following that is applicable:

• kind/feature
• kind/improvement
• kind/deprecation
• kind/documentation
• kind/clean-up
• kind/bug
• kind/other

Optional: Mark one or more of the following that are applicable:

Important

Breaking changes should be marked kind/admin-change or kind/dev-change depending on type
Critical security fixes should be marked with kind/security

• kind/admin-change
• kind/dev-change
• kind/security
• [kind/adr](set-me)

What does this PR do / why do we need this PR?

This PR enables Minio ingress and updates node-local-dns and common-config to support a shared objectstorage setup between SC and WC in the local cluster environment.
...

• Fixes [2] Ensure object storage on local clusters are reachable across sc and wc #2769

Information to reviewers

How to run / how to test.

Init config

./scripts/local-cluster.sh config <name> dev <domain-name>

Create the Service Cluster (SC) with ingress enabled:

./scripts/local-cluster.sh create <clustername>-sc <profile> 

Create the Workload Cluster (WC) without Minio:

./scripts/local-cluster.sh create <clustername>-wc <profile> --skip-minio

Set up NodeLocalDNS:

./scripts/local-cluster.sh setup node-local-dns

Install Velero in the WC to verify that storage from SC is accessible:

helmfile -e workload_cluster -lapp=velero apply

Once all Velero pods are running, check the backup location:

velero backup-location get 

Expected output:

velero backup-location get
NAME      PROVIDER   BUCKET/PREFIX                        PHASE       LAST VALIDATED                  ACCESS MODE   DEFAULT
default   aws        local-ck8s-velero/workload-cluster   Available   2025-11-27 11:53:48 +0100 CET   ReadWrite     true

--->

Checklist

• Proper commit message prefix on all commits
• Change checks:
  • The change is transparent
  • The change is disruptive
  • The change requires no migration steps
  • The change requires migration steps
  • The change updates CRDs
  • The change updates the config and the schema
• Documentation checks:
  • The public documentation required no updates
  • The public documentation required an update - [link to change](set-me)
• Metrics checks:
  • The metrics are still exposed and present in Grafana after the change
  • The metrics names didn't change (Grafana dashboards and Prometheus alerts required no updates)
  • The metrics names did change (Grafana dashboards and Prometheus alerts required an update)
• Logs checks:
  • The logs do not show any errors after the change
• PodSecurityPolicy checks:
  • Any changed Pod is covered by Kubernetes Pod Security Standards
  • Any changed Pod is covered by Gatekeeper Pod Security Policies
  • The change does not cause any Pods to be blocked by Pod Security Standards or Policies
• NetworkPolicy checks:
  • Any changed Pod is covered by Network Policies
  • The change does not cause any dropped packets in the NetworkPolicy Dashboard
• Audit checks:
  • The change does not cause any unnecessary Kubernetes audit events
  • The change requires changes to Kubernetes audit policy
• Falco checks:
  • The change does not cause any alerts to be generated by Falco
• Bug checks:
  • The bug fix is covered by regression tests

[shafi-elastisys]

I’ve added the setup so it can be triggered via a script whenever we need this functionality.
If we are fine with this approach, I’ll update the Development.md file with the corresponding instructions.

[aarnq]

Could we instead make this a feature available through the config, so you set it as an option during the configuration stage?

Also, this changes how ingress traffic is handled from the default case which I'm not sure we want here given that we have solved that issue before without using host ports.
As it would severely limit the capabilities of testing NetworkPolicies on local clusters for ingress traffic.

[simonklb]

I agree with @aarnq. This looks like a workaround patch to the setup. Is there something preventing this to be configured properly instead?

[shafi-elastisys]

Now I have changed the solution without enabling hostport in ingress. Also setup of shared object storage is added during the configuration stage.

[simonklb]

Why can't we just create a new config in https://github.com/elastisys/compliantkubernetes-apps/blob/main/scripts/local-clusters/configs or incorporate this in the existing config(s) instead of patching this afterwards?

[simonklb]

Can this not be set in config?

[shafi-elastisys]

we are also having task to make the dev more simpler. I think I will rework on the PR to make this as default configuration of setting up object storage sharable between SC and WC. So that the instruction is less also simple. Does that sound good?

[simonklb]

Sounds good to me. I don't see why you would not want a shared object storage between the clusters in the local cluster setup since that is closer to what we have in real environments.

[shafi-elastisys]

Yeah I would want it same way. That's why I mentioned it can be a default setup instead of making it optional setup using script.

[simonklb]

To be clear, when I wrote "I don't see why you would not want..." I meant you as in everyone generally, not you specifically. 😄

[shafi-elastisys]

Yeah I understood in that way only 😄