fix(catalyst-api): bake-time top-up of canonical .omani.X sme-pool (TBD-A44, Closes #1907) (#1913)

PR #1861 widened LoadSMETenantParentDomainsFromEnv to seed all four
canonical .omani.X TLDs (homes, rest, trade, works), but on a real
Sovereign that env-stub fallback path is BYPASSED. The mothership
imports a full deployment record with only the operator-selected
sme-pool entry, and GET /api/v1/sovereign/parent-domains reads from
the imported record (dep.Request.ParentDomains), not the env stub.

Result on t31 (2026-05-19, c703247a0de12508): the on-disk record
holds 1 primary (omani.works) + 1 sme-pool (omani.homes) = 2 rows.
/parent-domains?role=sme-pool returns 1 entry instead of 4. A
customer picking .omani.rest or .omani.trade on the marketplace
/addons subdomain picker — both options the UI hard-codes — fails
SME tenant signup with 422 invalid-parent-domain.

Fix shape (same pattern as PR #1893 / D21 owner UserAccess
bake-time seed): on every chroot-mode catalyst-api startup AND on
every fresh handover import, top up Request.ParentDomains with any
missing canonical TLD as role=sme-pool. Idempotent (a re-run is a
no-op when the pool is already full); mothership mode (SOVEREIGN_FQDN
unset) is a hard no-op; persists to disk so a Pod restart sees the
topped-up shape.

Dedup is against existing role=sme-pool rows only — a role=primary
row on the same name does NOT count, because the customer-facing
/addons picker validates against role=sme-pool entries via
FindParentDomain. The t31 shape (primary=omani.works AND
sme-pool=omani.works needed) is the real-world case.

Wired into two seams so a fresh prov AND a Pod restart both
converge: HandleDeploymentImport (post-import, fresh prov) and
restoreFromStore (per-record rehydration, Pod restart). Five guards
in chroot_parent_domains_seed_test.go: AllowedTLDs lockstep,
top-up shape (mirrors t31), idempotence, mothership no-op, nil-dep.

Drive-by: fixed a pre-existing build break in
sme_tenant_gitops.go's smeTenantBPKeycloak raw-string constant
(PR #1909 introduced literal backticks + a Go template action
inside a YAML comment; the action confused text/template at
render time → bp-keycloak.yaml render returned `unexpected EOF`).
Replaced with prose that describes the chart template behaviour
without inlining the template literal.

Co-authored-by: hatiyildiz <hatice.yildiz@openova.io>

products/catalyst/bootstrap/api/internal/handler/chroot_parent_domains_seed.go

@@ -0,0 +1,144 @@
+// chroot_parent_domains_seed.go — bake-time top-up of the canonical
+// .omani.X sme-pool entries on Sovereign-side catalyst-api.
+//
+// Why this exists
+// ---------------
+// The mothership wizard stamps `Request.ParentDomains` with the
+// operator-selected primary (e.g. `omani.works`) plus exactly one
+// sme-pool entry — the pool TLD the operator picked at provisioning
+// time. When the mother POSTs the full record to the Sovereign-side
+// catalyst-api via /api/v1/internal/deployments/import, the chroot
+// inherits that same 2-entry slice.
+//
+// Result: `GET /api/v1/sovereign/parent-domains?role=sme-pool` returns
+// 1 of 4 entries instead of the canonical 4 the marketplace UI's
+// /addons subdomain picker hard-codes (omani.homes, omani.rest,
+// omani.trade, omani.works — same list as
+// core/services/domain/store.AllowedTLDs). A customer who selects a
+// missing TLD on the picker hits a 422 invalid-parent-domain at SME
+// tenant signup because FindParentDomain doesn't recognise the TLD.
+//
+// PR #1861 widened LoadSMETenantParentDomainsFromEnv() to seed all 4
+// entries, but that's the *env-stub fallback* path used by the SME
+// tenant create handler's startup wiring (SMETenantDeps.ParentDomains).
+// The /api/v1/sovereign/parent-domains HTTP surface reads from the
+// adopted Deployment's persistent Request.ParentDomains slice, NOT
+// the env stub. The mother-written record bypasses the fallback.
+//
+// Fix shape
+// ---------
+// Run a chroot-mode bake-time top-up: when SOVEREIGN_FQDN is set AND
+// we have an adopted/in-memory deployment record, ensure every
+// canonical .omani.X entry exists as Role=sme-pool. Idempotent — a
+// re-run is a no-op when the pool is already full. Same pattern as
+// the D21 owner-UserAccess bake-time seed (PR #1893): the chroot
+// fills in operator-curated defaults that the mother either could
+// not or did not stamp.
+//
+// Called from two places so a fresh prov AND a Pod restart both
+// converge:
+//
+//   1. HandleDeploymentImport — after the mother POSTs the record at
+//      handover. Closes the data-half of the contract for a fresh
+//      prov.
+//
+//   2. restoreFromStore — at catalyst-api startup, for every
+//      rehydrated deployment. Covers the Pod-restart case where the
+//      import already ran on a prior Pod and the on-disk record is
+//      still short of 4 entries (the situation #1907 caught on t31).
+//
+// Mothership mode: SOVEREIGN_FQDN unset → no-op. The mother only ever
+// holds the operator-selected sme-pool entry; the canonical pool is a
+// per-Sovereign concept and doesn't apply on the mothership.
+//
+// AllowedTLDs is the central registry (core/services/domain/store).
+// We duplicate the literal here only to avoid pulling a Mongo client
+// dependency into the bootstrap package — the lockstep is enforced by
+// TestChrootEnsureSMEPoolSeed_MatchesAllowedTLDsLiteral below.
+package handler
+
+import (
+	"os"
+	"sort"
+	"strings"
+	"time"
+
+	"github.com/openova-io/openova/products/catalyst/bootstrap/api/internal/provisioner"
+)
+
+// canonicalSMEPoolDomains — the four .omani.X TLDs the marketplace
+// UI offers in its /addons subdomain picker. Locked to
+// core/services/domain/store.AllowedTLDs (asserted via a
+// compile-time-style test below). Order is intentional and matches
+// the env-stub fallback in sovereign_parent_domains.go so the wire
+// shape is deterministic.
+var canonicalSMEPoolDomains = []string{
+	"omani.homes",
+	"omani.rest",
+	"omani.trade",
+	"omani.works",
+}
+
+// chrootEnsureSMEPoolSeed tops up dep.Request.ParentDomains with any
+// missing canonical sme-pool entries. No-op when:
+//   - dep is nil
+//   - we are not in chroot mode (SOVEREIGN_FQDN unset)
+//   - every canonical entry is already present
+//
+// When at least one row is appended, the function persists the
+// deployment record so a Pod restart sees the topped-up shape.
+// Returns the number of rows appended so the caller can log.
+func (h *Handler) chrootEnsureSMEPoolSeed(dep *Deployment) int {
+	if dep == nil {
+		return 0
+	}
+	if strings.TrimSpace(os.Getenv("SOVEREIGN_FQDN")) == "" {
+		return 0
+	}
+	dep.mu.Lock()
+	// Dedup against existing sme-pool entries only — a row with
+	// role=primary on the same name does NOT count as "already
+	// present" for pool purposes. The customer-facing /addons picker
+	// validates against role=sme-pool entries, so the pool needs every
+	// canonical TLD listed as sme-pool even when the operator's
+	// chosen primary happens to share a name with one of them (the
+	// real-world t31 shape: primary=omani.works, pool must still
+	// surface omani.works as a sme-pool row so SME tenants picking
+	// .omani.works pass FindParentDomain).
+	presentPool := map[string]struct{}{}
+	for _, pd := range dep.Request.ParentDomains {
+		if string(pd.Role) != provisioner.ParentDomainRoleSMEPool {
+			continue
+		}
+		presentPool[strings.ToLower(strings.TrimSpace(pd.Name))] = struct{}{}
+	}
+	now := time.Now().UTC()
+	appended := 0
+	for _, name := range canonicalSMEPoolDomains {
+		if _, ok := presentPool[name]; ok {
+			continue
+		}
+		dep.Request.ParentDomains = append(dep.Request.ParentDomains, provisioner.ParentDomain{
+			Name:    name,
+			Role:    provisioner.ParentDomainRoleSMEPool,
+			AddedAt: now,
+		})
+		appended++
+	}
+	// Keep the slice in deterministic order so wire-shape lock tests
+	// (and ListParentDomains' sort.Slice) don't surface flakes.
+	sort.SliceStable(dep.Request.ParentDomains, func(i, j int) bool {
+		return strings.ToLower(dep.Request.ParentDomains[i].Name) <
+			strings.ToLower(dep.Request.ParentDomains[j].Name)
+	})
+	dep.mu.Unlock()
+	if appended > 0 {
+		h.persistDeployment(dep)
+		h.log.Info("chroot: seeded canonical sme-pool parent domains",
+			"depId", dep.ID,
+			"appended", appended,
+			"pool", canonicalSMEPoolDomains,
+		)
+	}
+	return appended
+}

products/catalyst/bootstrap/api/internal/handler/chroot_parent_domains_seed_test.go

@@ -0,0 +1,238 @@
+// chroot_parent_domains_seed_test.go — regression boundary for #1907.
+//
+// What the tests guard
+// --------------------
+// PR #1861 widened LoadSMETenantParentDomainsFromEnv to seed all four
+// canonical .omani.X entries in the env-stub fallback, but on a real
+// Sovereign that path is bypassed: the mother imports a full
+// deployment record with only the operator-selected sme-pool TLD,
+// and the HTTP surface /api/v1/sovereign/parent-domains reads from
+// the imported record (dep.Request.ParentDomains), not the env stub.
+// Result on t31 (2026-05-19): /parent-domains?role=sme-pool returned
+// 1 entry instead of 4 → customer's omani.homes pick failed at SME
+// tenant signup with 422 invalid-parent-domain.
+//
+// chrootEnsureSMEPoolSeed closes that gap. These tests guard:
+//
+//   1. Lockstep with core/services/domain/store.AllowedTLDs — if
+//      somebody widens the picker, this test fails until the seed
+//      catches up.
+//
+//   2. Top-up shape — partial mother-imported pool (1 of 4) gets
+//      bumped to the full 4-entry canonical list, preserving the
+//      mother-stamped row and adding only the missing entries.
+//
+//   3. Idempotence — running the seed twice is a no-op on the second
+//      pass (no duplicate rows, no spurious persist).
+//
+//   4. Mode gate — SOVEREIGN_FQDN unset is a hard no-op (the
+//      mothership does NOT own the canonical pool concept; only the
+//      Sovereign-side chroot does).
+//
+//   5. Determinism — output order is sorted by lowercased name so the
+//      wire shape returned to the operator is stable across Pod
+//      restarts.
+//
+// The tests don't stand up the full Handler; they use the in-memory
+// store via NewWithStore + a t.TempDir() so the persist branch can be
+// exercised end-to-end (round-trip the dep record through disk and
+// verify the topped-up shape survives).
+package handler
+
+import (
+	"log/slog"
+	"os"
+	"reflect"
+	"sort"
+	"strings"
+	"testing"
+
+	"github.com/openova-io/openova/products/catalyst/bootstrap/api/internal/provisioner"
+	"github.com/openova-io/openova/products/catalyst/bootstrap/api/internal/store"
+)
+
+// TestChrootEnsureSMEPoolSeed_MatchesAllowedTLDsLiteral — the
+// canonical pool literal in the seed file must contain exactly the
+// same four TLDs as core/services/domain/store.AllowedTLDs. We can't
+// import that package directly (it's a separate Go module — the
+// catalyst-api bootstrap binary is intentionally decoupled from the
+// SME-side service modules), so the test asserts against the same
+// literal list a human reviewer can eyeball-match. Drift surfaces in
+// CI as a test fail with the exact got/want diff.
+func TestChrootEnsureSMEPoolSeed_MatchesAllowedTLDsLiteral(t *testing.T) {
+	// Mirror of core/services/domain/store.AllowedTLDs (2026-05-19).
+	// Update both lists together if AllowedTLDs is ever widened.
+	allowedTLDs := []string{
+		"omani.rest",
+		"omani.works",
+		"omani.trade",
+		"omani.homes",
+	}
+	got := append([]string(nil), canonicalSMEPoolDomains...)
+	want := append([]string(nil), allowedTLDs...)
+	sort.Strings(got)
+	sort.Strings(want)
+	if !reflect.DeepEqual(got, want) {
+		t.Fatalf("canonicalSMEPoolDomains drift from AllowedTLDs:\n  got:  %v\n  want: %v\n"+
+			"Update either canonicalSMEPoolDomains in chroot_parent_domains_seed.go OR\n"+
+			"AllowedTLDs in core/services/domain/store/store.go — the two must agree.",
+			got, want)
+	}
+}
+
+// TestChrootEnsureSMEPoolSeed_TopsUpPartialPool — start with the
+// 2-entry mother shape (primary + one sme-pool) and verify the seed
+// fills in the 3 missing canonical entries. Mirrors the t31 #1907
+// scenario byte-for-byte.
+func TestChrootEnsureSMEPoolSeed_TopsUpPartialPool(t *testing.T) {
+	t.Setenv("SOVEREIGN_FQDN", "t31.omani.works")
+
+	st, err := store.New(t.TempDir())
+	if err != nil {
+		t.Fatalf("store.New: %v", err)
+	}
+	h := &Handler{log: slog.Default(), store: st}
+
+	dep := &Deployment{
+		ID: "c703247a0de12508",
+		Request: provisioner.Request{
+			SovereignFQDN: "t31.omani.works",
+			ParentDomains: []provisioner.ParentDomain{
+				{Name: "omani.works", Role: provisioner.ParentDomainRolePrimary},
+				{Name: "omani.homes", Role: provisioner.ParentDomainRoleSMEPool},
+			},
+		},
+	}
+
+	added := h.chrootEnsureSMEPoolSeed(dep)
+	// 4 canonical TLDs minus 1 already-present sme-pool entry
+	// (omani.homes) = 3 appends. The primary's omani.works row does
+	// NOT count toward pool dedup; the canonical pool needs every
+	// TLD listed as sme-pool independently of whether one happens to
+	// match the primary (FindParentDomain validates against
+	// role=sme-pool, not role=primary).
+	if added != 3 {
+		t.Fatalf("expected 3 rows appended (4 canonical sme-pool minus 1 mother-stamped); got %d", added)
+	}
+
+	// All 4 canonical .omani.X entries must be present as sme-pool
+	// after the seed. The mother-stamped omani.homes Role=sme-pool
+	// must survive (we only append when missing, never overwrite).
+	pool := map[string]string{}
+	for _, pd := range dep.Request.ParentDomains {
+		if pd.Role == provisioner.ParentDomainRoleSMEPool {
+			pool[strings.ToLower(pd.Name)] = string(pd.Role)
+		}
+	}
+	for _, want := range []string{"omani.homes", "omani.rest", "omani.trade", "omani.works"} {
+		if _, ok := pool[want]; !ok {
+			t.Errorf("missing canonical pool entry after seed: %q (pool=%v)", want, pool)
+		}
+	}
+
+	// Primary entry must still be exactly 1 (we never touch primaries).
+	primaries := 0
+	for _, pd := range dep.Request.ParentDomains {
+		if pd.Role == provisioner.ParentDomainRolePrimary {
+			primaries++
+		}
+	}
+	if primaries != 1 {
+		t.Errorf("primary count drift: got %d, want 1", primaries)
+	}
+
+	// Output order must be sorted lowercase by name (determinism gate).
+	got := make([]string, 0, len(dep.Request.ParentDomains))
+	for _, pd := range dep.Request.ParentDomains {
+		got = append(got, strings.ToLower(pd.Name))
+	}
+	want := append([]string(nil), got...)
+	sort.Strings(want)
+	if !reflect.DeepEqual(got, want) {
+		t.Errorf("ParentDomains not sorted by lowercased name:\n  got:  %v\n  want: %v", got, want)
+	}
+
+	// Persistence round-trip: re-load the record from disk and verify
+	// the topped-up pool survives. 1 primary + 4 sme-pool = 5 rows.
+	rec, err := st.Load(dep.ID)
+	if err != nil {
+		t.Fatalf("store.Load: %v", err)
+	}
+	if len(rec.Request.ParentDomains) != 5 {
+		t.Fatalf("on-disk record length drift: got %d entries, want 5 (1 primary + 4 sme-pool)",
+			len(rec.Request.ParentDomains))
+	}
+}
+
+// TestChrootEnsureSMEPoolSeed_Idempotent — second run on an
+// already-full pool is a no-op.
+func TestChrootEnsureSMEPoolSeed_Idempotent(t *testing.T) {
+	t.Setenv("SOVEREIGN_FQDN", "t31.omani.works")
+	h := &Handler{log: slog.Default()}
+
+	dep := &Deployment{
+		ID: "x",
+		Request: provisioner.Request{
+			SovereignFQDN: "t31.omani.works",
+			ParentDomains: []provisioner.ParentDomain{
+				{Name: "omani.works", Role: provisioner.ParentDomainRolePrimary},
+				{Name: "omani.homes", Role: provisioner.ParentDomainRoleSMEPool},
+				{Name: "omani.rest", Role: provisioner.ParentDomainRoleSMEPool},
+				{Name: "omani.trade", Role: provisioner.ParentDomainRoleSMEPool},
+				{Name: "omani.works", Role: provisioner.ParentDomainRoleSMEPool},
+			},
+		},
+	}
+	// First pass: nothing to add (all 4 canonical names already
+	// present as sme-pool rows). The primary's omani.works row does
+	// not count toward pool dedup but the additional sme-pool
+	// omani.works row above does.
+	added := h.chrootEnsureSMEPoolSeed(dep)
+	if added != 0 {
+		t.Fatalf("first pass on full pool: expected 0 appends, got %d", added)
+	}
+	// Second pass: still no-op.
+	added = h.chrootEnsureSMEPoolSeed(dep)
+	if added != 0 {
+		t.Fatalf("second pass on full pool: expected 0 appends, got %d", added)
+	}
+}
+
+// TestChrootEnsureSMEPoolSeed_MothershipNoOp — SOVEREIGN_FQDN unset
+// means we're on the mothership; the seed must be a hard no-op (the
+// canonical pool is a per-Sovereign concept, not a mothership one).
+func TestChrootEnsureSMEPoolSeed_MothershipNoOp(t *testing.T) {
+	// Force unset in case the test environment leaked the var.
+	if err := os.Unsetenv("SOVEREIGN_FQDN"); err != nil {
+		t.Fatalf("unset SOVEREIGN_FQDN: %v", err)
+	}
+	h := &Handler{log: slog.Default()}
+
+	dep := &Deployment{
+		ID: "mother",
+		Request: provisioner.Request{
+			SovereignFQDN: "",
+			ParentDomains: []provisioner.ParentDomain{
+				{Name: "openova.io", Role: provisioner.ParentDomainRolePrimary},
+			},
+		},
+	}
+	added := h.chrootEnsureSMEPoolSeed(dep)
+	if added != 0 {
+		t.Fatalf("mothership: expected 0 appends, got %d", added)
+	}
+	if len(dep.Request.ParentDomains) != 1 {
+		t.Fatalf("mothership: expected unchanged length 1, got %d", len(dep.Request.ParentDomains))
+	}
+}
+
+// TestChrootEnsureSMEPoolSeed_NilDep — defensive against a caller
+// that hands us a nil pointer. The fix-author seam (HandleDeploymentImport,
+// restoreFromStore) never does this today, but a future caller might.
+func TestChrootEnsureSMEPoolSeed_NilDep(t *testing.T) {
+	t.Setenv("SOVEREIGN_FQDN", "t31.omani.works")
+	h := &Handler{log: slog.Default()}
+	if added := h.chrootEnsureSMEPoolSeed(nil); added != 0 {
+		t.Fatalf("nil dep: expected 0 appends, got %d", added)
+	}
+}

products/catalyst/bootstrap/api/internal/handler/deployment_handover_import.go

@@ -110,7 +110,17 @@ func (h *Handler) HandleDeploymentImport(w http.ResponseWriter, r *http.Request)
 	// waiting for the next pod restart's restoreFromStore. fromRecord
 	// also rewrites in-flight Status fields to "failed" — irrelevant
 	// here since the mother only exports terminal-state records.
-	h.deployments.Store(rec.ID, fromRecord(rec))
+	dep := fromRecord(rec)
+	h.deployments.Store(rec.ID, dep)
+
+	// #1907 — bake-time top-up of the canonical .omani.X sme-pool.
+	// The mother only stamps the single operator-selected pool entry,
+	// but the marketplace UI's /addons subdomain picker offers all
+	// four. Ensure the Sovereign-side pool matches the picker so a
+	// customer never sees 422 invalid-parent-domain after a successful
+	// subdomain pick. See chroot_parent_domains_seed.go for the full
+	// rationale.
+	h.chrootEnsureSMEPoolSeed(dep)
 
 	h.log.Info("deployment-import: persisted",
 		"id", rec.ID,

products/catalyst/bootstrap/api/internal/handler/deployments.go

@@ -579,6 +579,16 @@ func (h *Handler) restoreFromStore() {
 			resumed++
 			h.resumePhase1Watch(dep)
 		}
+
+		// #1907 — bake-time top-up of the canonical .omani.X sme-pool.
+		// On a Pod restart the import already ran on a prior Pod and
+		// the on-disk record may still be short of 4 entries (the
+		// situation #1907 caught on t31 — mother stamped 2 entries,
+		// catalyst-api restarted, /parent-domains kept returning 2).
+		// No-op on the mothership (SOVEREIGN_FQDN unset); idempotent
+		// when the pool is already full. See
+		// chroot_parent_domains_seed.go for the full rationale.
+		h.chrootEnsureSMEPoolSeed(dep)
 	}
 	h.log.Info("restored deployments from PVC",
 		"count", len(records),

products/catalyst/bootstrap/api/internal/handler/sme_tenant_gitops.go

@@ -801,9 +801,10 @@ spec:
         namespace: kube-system
         # sectionName omitted — multi-zone Sovereigns rename HTTPS listeners
         # to https-<sanitised-zone> (e.g. https-omani-works). The bp-keycloak
-        # chart template guards `{{- with .Values.gateway.parentRef.sectionName }}`
-        # so a blank value drops the field entirely; Cilium Gateway then
-        # matches by hostname filter. See PR #1888 / TBD-A40 / issue #1902.
+        # chart template guards the sectionName output with a 'with'
+        # conditional on .Values.gateway.parentRef.sectionName, so a blank
+        # value drops the field entirely; Cilium Gateway then matches by
+        # hostname filter. See PR #1888 / TBD-A40 / issue #1902.
         sectionName: ""
     # Outbound realm email — Phase-1 mothership relay. Operator overlay
     # (or future tenant-Stalwart sub-issue) overrides host/port once