-
Notifications
You must be signed in to change notification settings - Fork 159
Shared Multi Step Form for Object Storage Connectors #8467
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
lovincyrus
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review: Shared Multi Step Form for Object Storage Connectors
Overview
This PR implements a shared, config-driven multi-step form architecture for object storage connectors (GCS, S3, Azure). It replaces a GCS-specific implementation with a generalized system that splits connector configuration into two steps:
- Connector step: Authentication configuration
- Source step: Resource path and model creation
The refactoring demonstrates good abstraction design by extracting authentication patterns into reusable configurations.
Strengths
1. Excellent Abstraction & Reusability ✅
The introduction of multi-step-auth-configs.ts is a strong design decision. This configuration-driven approach allows adding new connectors without duplicating form logic. The config structure is well-designed with:
authOptions: Defines available authentication methodsauthFieldGroups: Maps each auth method to its required fieldsclearFieldsByMethod: Ensures form state hygiene when switching auth methodsexcludedKeys: Prevents duplicate rendering of auth fields
2. Strong Type Safety ✅
The type definitions in types.ts are well-structured with discriminated unions for AuthField ensuring type-safe rendering based on field type.
3. Good Component Boundaries ✅
The component hierarchy is well-organized:
AddDataForm.svelte (orchestrator)
↓
MultiStepFormRenderer.svelte (config → component adapter)
↓
MultiStepAuthForm.svelte (auth UI logic)
Each component has a clear responsibility.
Areas for Improvement
1. State Management Complexity ⚠️
Issue: The selectedAuthMethod state is managed across multiple layers with initialization logic scattered in AddDataForm.svelte:313-327. Multiple reactive statements compete to set the value, passed through multiple components with bind:authMethod.
Problems:
- Initialization race conditions: Multiple reactive statements compete to set
selectedAuthMethod - Unclear ownership: Who owns this state? Parent or child?
- Difficult debugging: State changes happen reactively across files
Recommendation: Centralize auth method state in connectorStepStore:
// connectorStepStore.ts
export const connectorStepStore = writable<{
step: ConnectorStep;
connectorConfig: Record<string, unknown> | null;
selectedAuthMethod: string | null; // Add this
}>({
step: "connector",
connectorConfig: null,
selectedAuthMethod: null,
});
export function setAuthMethod(method: string) {
connectorStepStore.update((state) => ({ ...state, selectedAuthMethod: method }));
}2. Dynamic Validation Logic Needs Improvement ⚠️
The validation in FormValidation.ts:50-86 uses test-based conditional validation with getter dependencies.
Problems:
- Fragile getter dependency: Relies on
authMethodGetterfunction passed through constructor - Validation state not obvious: Hard to understand which fields are validated when
Recommendation: Use yup.when() with explicit dependencies:
fieldValidations[field.id] = yup.string().when('$authMethod', {
is: (authMethod: string) => authMethod === method,
then: (schema) => schema.required(`${label} is required`),
otherwise: (schema) => schema.optional(),
});3. Submit Button Logic Too Complex ⚠️
The isSubmitDisabled calculation in AddDataForm.svelte:137-210 is ~74 lines with nested conditionals.
Problems:
- Hard to test: Giant reactive statement with multiple branches
- Hard to maintain: Adding new conditions requires understanding all existing logic
- Duplicates validation logic: Re-implements field requirement checks
Recommendation: Extract to well-named functions:
function isMultiStepConnectorDisabled(config, selectedAuthMethod, paramsForm, paramsErrors) {
const method = selectedAuthMethod || config.defaultAuthMethod || config.authOptions?.[0]?.value;
if (!method) return true;
const fields = config.authFieldGroups?.[method] || [];
if (!fields.length && method === 'public') return false;
return !fields.every(field => {
if (field.optional ?? false) return true;
const value = paramsForm[field.id];
return !isEmpty(value) && !paramsErrors[field.id]?.length;
});
}
$: isSubmitDisabled = (() => {
if (isMultiStepConnector && stepState.step === "connector") {
return isMultiStepConnectorDisabled(activeMultiStepConfig, selectedAuthMethod, $paramsForm, $paramsErrors);
}
// ... other cases
})();4. Form State Clearing Logic Scattered ⚠️
Field clearing happens reactively in MultiStepAuthForm.svelte:28-39. If someone forgets to update clearFieldsByMethod, stale data persists.
Recommendation: Make clearing explicit and comprehensive:
function clearUnusedAuthFields(
form: Record<string, any>,
config: MultiStepFormConfig,
selectedMethod: string
): Record<string, any> {
const allAuthFields = new Set(
Object.values(config.authFieldGroups).flat().map(f => f.id)
);
const activeFields = new Set(
config.authFieldGroups[selectedMethod]?.map(f => f.id) || []
);
const clearedForm = { ...form };
for (const field of allAuthFields) {
if (!activeFields.has(field)) {
clearedForm[field] = "";
}
}
return clearedForm;
}This approach clears all non-active auth fields automatically and doesn't require maintaining clearFieldsByMethod separately.
5. Submit Flow Has Hidden Public Auth Shortcut ⚠️
In AddDataFormManager.ts:317-321, public auth skips connection testing:
if (selectedAuthMethod === "public") {
setConnectorConfig(values);
setStep("source");
return;
}Problems:
- Hidden logic: User clicks "Continue" but validation still runs
- Hard to discover: This shortcut is buried in the submit handler
Recommendation: Extract to named functions:
private handlePublicAuthSubmit(values: Record<string, unknown>) {
// Public auth doesn't require connection testing
setConnectorConfig(values);
setStep("source");
}
private async handlePrivateAuthSubmit(queryClient: any, values: Record<string, unknown>) {
await submitAddConnectorForm(queryClient, this.connector, values, false);
setConnectorConfig(values);
setStep("source");
}6. "Save Anyway" Logic is Confusing ⚠️
The conditions for showing "Save Anyway" button in AddDataFormManager.ts:287-307 have 6 conditions ANDed together.
Recommendation: Extract to a well-named predicate:
private shouldShowSaveAnywayButton(args: {
isConnectorForm: boolean;
event: any;
stepState: ConnectorStepState;
selectedAuthMethod?: string;
}): boolean {
const { isConnectorForm, event, stepState, selectedAuthMethod } = args;
// Only show for connector forms (not sources)
if (!isConnectorForm) return false;
// ClickHouse has its own error handling
if (this.connector.name === "clickhouse") return false;
// Need a submission result to show the button
if (!event?.result) return false;
// Multi-step connectors: don't show on source step (final step)
if (stepState?.step === "source") return false;
// Public auth bypasses connection test, so no "Save Anyway" needed
if (stepState?.step === "connector" && selectedAuthMethod === "public") return false;
return true;
}7. Backend Connector Property Changes Not Well Explained ⚠️
The backend changes in runtime/drivers/s3/s3.go remove role assumption properties (aws_role_arn, aws_role_session_name, aws_external_id).
Problems:
- Breaking change: Existing connectors using role assumption will break
- No migration path: PR description doesn't mention how to handle existing configs
Recommendation:
- Add migration guide in PR description
- Consider deprecation path
- Add comment explaining why removed
8. Magic String "public" Used Throughout ⚠️
The string "public" appears in multiple places without constants.
Recommendation: Define constants:
// constants.ts
export const AUTH_METHOD_PUBLIC = "public" as const;
export const AUTH_METHOD_CREDENTIALS = "credentials" as const;
// Usage:
if (selectedAuthMethod === AUTH_METHOD_PUBLIC) { ... }9. UI Copy Should Be Extracted to Constants ⚠️
Button labels are hardcoded in AddDataFormManager.ts:229-247.
Recommendation: Extract to a config:
export const BUTTON_LABELS = {
public: { idle: "Continue", submitting: "Continuing..." },
connector: { idle: "Test and Connect", submitting: "Testing connection..." },
source: { idle: "Import Data", submitting: "Importing data..." },
} as const;10. Adding New Connectors Requires Updates in Multiple Files ⚠️
To add a new multi-step connector, you need to update 4 different files.
Recommendation: Enforce via types:
// Ensure every MULTI_STEP_CONNECTOR has a config
type MultiStepConnectorName = typeof MULTI_STEP_CONNECTORS[number];
const _typeCheck: Record<MultiStepConnectorName, MultiStepFormConfig> = multiStepFormConfigs;This creates a compile-time error if someone adds a connector without a config.
Testing Concerns
11. Complex Logic Needs More Tests ⚠️
Key areas needing tests:
- Auth method switching: Does field clearing work correctly?
- Validation logic: Are required fields enforced per auth method?
- Public auth shortcut: Does it skip connection testing correctly?
- Multi-step navigation: Can users go back? Is state preserved?
Summary & Recommendations
Priority Fixes (Before Merge)
⚠️ Centralize auth method state inconnectorStepStoreto avoid initialization races⚠️ ExtractisSubmitDisabledlogic to testable functions⚠️ Add constants for magic strings ("public", etc.)⚠️ Document backend breaking changes and provide migration path
Medium Priority (Can Be Follow-up)
- Improve validation with
yup.when()instead of test-based validation - Simplify "Save Anyway" logic with named predicates
- Extract field clearing logic to be config-driven
- Add comprehensive tests for auth switching and validation
Low Priority (Nice to Have)
- Extract button labels to constants
- Standardize error handling utilities
- Add type-level enforcement for multi-step connector configs
Final Verdict
Overall Assessment: Good abstraction work with some state management complexity
- ✅ Readability: Good component boundaries, but some overly complex reactive statements
- ✅ Maintainability: Config-driven design is excellent, but state management needs simplification
⚠️ Abstractions: Strong config system, but validation and state management could be cleaner⚠️ Function Boundaries: Good separation, but some functions are too large (isSubmitDisabled)⚠️ State Management: Needs improvement - auth method state scattered across components
The PR successfully generalizes the multi-step form pattern, making it reusable for all object storage connectors. The main concern is state management complexity around selectedAuthMethod, which should be centralized in the store before merging.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
GCS: https://jam.dev/c/74536bbf-e99d-4381-b4ad-f9afd6b007a4
- when switching between JSON and HMAC, there's a split sec where HMAC shows missing creds in red
- when skipping using "already connected?" I cannot add a model, the button flickers but does nothing (see end of recording)
S3: https://jam.dev/c/1f2b5a0d-2ba3-46c4-95fb-73ce3d93b459
- connectors are iterating s3, s4, s5 intead of s3, s3_1, s3_2
- same as above: when skipping using "already connected?" I cannot add a model, the button flickers but does nothing
Azure: https://jam.dev/c/a9dfe375-4ed9-4c8a-9168-0451e2894ab2
- same as above: when skipping using "already connected?" I cannot add a model, the button flickers but does nothing
Overall good! Thanks for adding the additional changes that I messaged you with!
Fixed! @royendo |
royendo
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have migrated the fix from #8354 |
royendo
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good from Product QA.
|
Hey Cyrus, A while back we wrote a design doc for connector forms that aligned on using JSON Schema as the form definition format: https://www.notion.so/rilldata/Snippets-API-design-doc-1f5ba33c8f5780798a0cfc819a406b42 The idea is that these schemas eventually live on the backend and can be used by both UI and AI/CLI. Defining a config-driven form system that works across multiple connectors is definitely the right direction! But the custom TypeScript format ( Here's a rough sketch of what S3 might look like: {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"auth_method": {
"type": "string",
"title": "Authentication method",
"enum": ["access_keys", "public"],
"default": "access_keys",
"x-display": "radio"
},
"aws_access_key_id": {
"type": "string",
"title": "Access Key ID",
"description": "AWS access key ID for the bucket",
"x-secret": true,
"x-step": "connector",
"x-visible-if": { "auth_method": "access_keys" }
},
"aws_secret_access_key": {
"type": "string",
"title": "Secret Access Key",
"description": "AWS secret access key for the bucket",
"x-secret": true,
"x-step": "connector",
"x-visible-if": { "auth_method": "access_keys" }
},
"path": {
"type": "string",
"title": "S3 URI",
"pattern": "^s3://",
"x-step": "source"
},
"name": {
"type": "string",
"title": "Model name",
"pattern": "^[a-zA-Z_][a-zA-Z0-9_]*$",
"x-step": "source"
}
},
"allOf": [
{
"if": { "properties": { "auth_method": { "const": "access_keys" } } },
"then": { "required": ["aws_access_key_id", "aws_secret_access_key"] }
}
]
}This is illustrative—you'll need to think through the right JSON Schema properties. Try to stay as close to native JSON Schema as possible before reaching for custom
There may be better ways to model some of these. Let me know if you want to talk through it. Developed in collaboration with Claude Code |
ericpgreen2
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See the comment above. A declarative config language to define these forms is definitely the right direction! It's so close to the JSON Schema approach we had discussed previously. This seems like a good time to take your code here one step closer to that design doc.
Added the json schema changes in 83bfcdf |
|
Nice work adopting JSON Schema—this is the right direction. One structural concern: the current flow is A cleaner approach would be a Keep the renderer generic—use slots for context-specific UI outside the form fields themselves: the "Already connected? Import your data" link, connector-specific error messages, etc. That way the same renderer can be reused for non-connector templates later. For incremental migration, isolate the new pattern in its own top-level feature directory to align with the design doc: This keeps the new approach cleanly separated and positions it as a general pattern for template-based code generation, not just source connectors. Optionally, you could also extract the file generation logic from // Today: implemented client-side
// Tomorrow: calls RuntimeService.GenerateFile
async function generateFile(args: {
template: string;
properties: Record<string, unknown>;
preview?: boolean;
}): Promise<{
addedPaths: string[];
modifiedPaths: string[];
previews: Record<string, string>; // path → YAML
}>This would make the backend migration a simple swap—same interface, different implementation. Let me know if you want to talk through any of this. Developed in collaboration with Claude Code |
ericpgreen2
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See above feedback on the JSON Schema implementation
|
Good progress on the One more structural issue: the renderer still has connector-specific logic baked in:
This makes it a "connector form renderer" rather than a generic "JSON Schema form renderer." The principle: The renderer should be a pure function of the schema. It shouldn't know about connectors, sources, or auth methods. It should just:
The auth method pattern (radio with nested fields) is just one instance of a general pattern: "enum field with conditionally visible related fields." The renderer should handle that generically. Connector-specific logic belongs in a parent component. Some naming options:
This parent component handles:
The renderer just renders fields; the parent orchestrates the workflow. Directory structure: The The generic rendering infrastructure lives in Let me know if you want to talk through any of this. Developed in collaboration with Claude Code |
Address this round of feedback. The multi-step connectors now only depend on the schema-driven design, with the validation pattern strictly enforced. It is not trivial to migrate to schemasafe in this branch. We should perform this migration in a follow-up to align the current json schema with schemasafe. |
ericpgreen2
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for addressing the feedback. The cleanup looks good - the dead code removal, SchemaField rename/relocation, and isStepMatch move all look correct.
Regarding the validation architecture: I'd prefer we implement schemasafe before merging rather than deferring to a follow-up. The concern is that the translation layer in FormValidation.ts (~150 lines manually reimplementing what schemasafe handles) becomes tech debt that tends to linger. Your comment about needing to "align the current json schema with schemasafe" also suggests there may be schema adjustments required - those are easier to make now than after this is on main.
Suggestion: Create a new branch off this one and implement the schemasafe migration there. We can review both together and merge once the architecture is clean. If you hit genuine blockers with schemasafe, we can reassess at that point.
One additional item: SchemaField.svelte still imports from sources/modal:
import { normalizeErrors } from "../sources/modal/utils";This creates a dependency from generic templates/ code back to connector-specific code. normalizeErrors is a generic utility and should be moved within templates/.
Developed in collaboration with Claude Code
Okay, addressed both! I managed to add a schemasafe-based helper in |


This PR implements a shared, config-driven multi-step form architecture for object storage connectors (GCS, S3,
Azure). It replaces a GCS-specific implementation with a generalized system that splits connector configuration
into two steps:
Checklist: