PRODUCT draft: feat/connector schemas

[royendo]

This is a Product branch of expected behavior built on top of Cyrus' open PR for migration for cloud object storage

Main Changes:

• Full migration to .ts for connector UI
• CH Specific SVELTE and logic removed,
• CH Cloud and CH split
• readwrite and read radio icons added
• Advanced Config added (to support connector edits)
• Dropdown for mysql, pg ssl-mode
• https front end connector changes
• Snowflake Key-pair Auth
• Data Explorer for read only OLAP

Few bugs need to be fixed:

• Test and Connect not working for CHC user/pass
• Test and Connect not working for Clickhouse
• Need to make sure advanced configs works for all connectors
• e2e testing reqs
• radio isnt being selected
• some preview items not being shown

[royendo]

https://www.notion.so/rilldata/Figjam-Links-2d2ba33c8f578061a173d024b4eb44e0?source=copy_link#2d2ba33c8f5780a39201f730f7676a99

[royendo]

---

Phase 1: TypeScript / Schema Layer

Sprint 1A: Core Types & Utilities

1. Types - schemas/types.ts
   - JSONSchemaField with all x-* extensions
   - MultiStepFormSchema
   - Conditional types (allOf, if/then/else)
2. Schema utilities - schema-utils.ts
   - isVisibleForValues() - conditional visibility
   - findRadioEnumKey() / getRadioEnumOptions() - auth method detection
   - getRequiredFieldsByEnumValue() - dynamic required fields
   - findGroupedEnumKeys() - fields to exclude from YAML output
3. Connector schema registry - connector-schemas.ts
   - multiStepFormSchemas map
   - getConnectorSchema() lookup
   - isStepMatch() helper

---

Sprint 1B: Object Storage Schemas

schemas/s3.ts
schemas/gcs.ts
schemas/azure.ts

• Auth method variations (keys, tokens, public, IAM)
• x-grouped-fields for auth method → child fields mapping
• x-visible-if conditions

---

Sprint 1C: Relational Database Schemas

schemas/postgres.ts
schemas/mysql.ts
schemas/https.ts

• Connection string vs parameters pattern
• x-display: "tabs" for connection method toggle

---

Sprint 1D: Cloud Warehouse Schemas

schemas/snowflake.ts
schemas/bigquery.ts
schemas/redshift.ts
schemas/athena.ts

• More complex auth (key pairs, service accounts)
• allOf conditional required fields

---

Sprint 1E: OLAP Engine Schemas (3-step)

schemas/clickhouse.ts
schemas/clickhouse-cloud.ts
schemas/duckdb.ts
schemas/motherduck.ts
schemas/druid.ts
schemas/pinot.ts

• mode: "read" | "readwrite" field
• x-step: "explorer" for read-only table selection
• x-step: "source" for read-write SQL/model

---

Sprint 1F: Validation Layer

1. Yup schemas Removal - yupSchemas.ts
2. Form validation - FormValidation.ts
   - Integrate JSON schema with existing validation
   - Step-aware validation
3. Source utilities - sourceUtils.ts
   - getInitialFormValuesFromProperties()
   - Schema-to-form-values mapping

---

Sprint 1G: Form Manager Updates

• AddDataFormManager.ts
  • Step-aware button labels
  • Schema-driven submission logic
  • getPrimaryButtonLabel() with mode awareness
• submitAddDataForm.ts
  • Handle 3-step submission flow
  • Connector config persistence between steps
• connectorStepStore.ts
  • Store shape for multi-step state
  • setAuthMethod(), step transitions

---

Phase 2: UI / Svelte Components

Sprint 2A: Form Renderer

• JSONSchemaFormRenderer.svelte
  • Renders fields by x-step
  • Handles x-display variants (radio, tabs, select, textarea, file)
  • x-visible-if reactive filtering
  • x-grouped-fields nesting
  • x-advanced collapsible section

---

Sprint 2B: Multi-Step Flow Component

• MultiStepConnectorFlow.svelte
  • Orchestrates connector → source → explorer steps
  • Auth method switching with form reset
  • Binds to form manager for button labels/state
• JSONSchemaFieldControl.svelte
  • Individual field rendering component

---

Sprint 2C: Modal Integration

• AddDataModal.svelte
  • Route to schema-based vs legacy forms
  • Handle explorer step for read-only connectors
• AddDataForm.svelte
  • Integrate MultiStepConnectorFlow
  • Step indicator UI
• AddDataExplorer.svelte
  • Table selection for explorer step
  • "Create metrics with AI" trigger

---

Sprint 2D: Cleanup

• Delete legacy components:
  • AddClickHouseForm.svelte
  • GCSMultiStepForm.svelte
  • Any hardcoded connector forms
• Remove connector-specific branching in shared components
• Update icon mapping if needed (connector-icon-mapping.ts)

---

Sprint 2E: Testing & Polish

• test-connection.spec.ts updates
• E2E for multi-step flows
• Visual regression for form layouts

---

This way you can:

1. Ship all schemas and get them reviewed independently
2. Test validation logic without UI changes
3. Then layer in UI with confidence the data layer is soli