Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(dw-edit-managed-sources): Edit Self Managed Sources for DW #28111

Merged
merged 27 commits into from
Feb 7, 2025
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
6cf654a
Setting up a new page for managing a self-managed source
phixMe Jan 29, 2025
f9a5ef4
Form is submitting.
phixMe Jan 30, 2025
6fb3d6d
Form resetting and resolving form issues.
phixMe Jan 30, 2025
3e7eada
Fixing error, removing some unused code, and improving readability.
phixMe Jan 30, 2025
1113676
Unused code removal and cache invalidation.
phixMe Jan 30, 2025
e2d2455
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Jan 30, 2025
dd5ee1a
Auto code review updates.
phixMe Jan 30, 2025
95c6862
Removing keys from update for now.
phixMe Jan 31, 2025
e33e6ff
Override update method.
phixMe Jan 31, 2025
c366540
Code review comments.
phixMe Feb 3, 2025
d2f8058
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 3, 2025
6886dd8
Update UI snapshots for `chromium` (2)
github-actions[bot] Feb 3, 2025
07e9e32
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 3, 2025
56a4b03
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 3, 2025
59511d8
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 4, 2025
7d8a233
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 5, 2025
d9a9849
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 5, 2025
cd97dbe
Kea logic colliding remediation
phixMe Feb 6, 2025
24d8e2c
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 6, 2025
f3e075c
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 6, 2025
68b1925
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 6, 2025
be122ed
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 7, 2025
8669142
Update UI snapshots for `chromium` (2)
github-actions[bot] Feb 7, 2025
9e62e77
data_warehouse --> self_managed
phixMe Feb 7, 2025
61237e4
Merge remote-tracking branch 'origin/feature(dw-edit-managed-sources)…
phixMe Feb 7, 2025
1eb4bee
Parity across managed and self-managed visuals.
phixMe Feb 7, 2025
fc3b2bf
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 80 additions & 49 deletions frontend/src/scenes/data-warehouse/new/DataWarehouseTableForm.tsx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { LemonInput, LemonSelect, Link } from '@posthog/lemon-ui'
import { LemonButton, LemonInput, LemonSelect, Link } from '@posthog/lemon-ui'
import { useValues } from 'kea'
import { Form } from 'kea-forms'
import { LemonField } from 'lib/lemon-ui/LemonField'
Expand Down Expand Up @@ -44,7 +44,11 @@ const ProviderMappings: Record<
},
}

export function DatawarehouseTableForm(): JSX.Element {
interface Props {
onUpdate?: () => void
}

export function DatawarehouseTableForm({ onUpdate }: Props): JSX.Element {
const { manualLinkingProvider } = useValues(sourceWizardLogic)

const provider = manualLinkingProvider ?? 'aws'
Expand All @@ -59,65 +63,85 @@ export function DatawarehouseTableForm(): JSX.Element {
>
<div className="flex flex-col gap-2 max-w-160">
<LemonField name="name" label="Table name">
<LemonInput
data-attr="table-name"
className="ph-ignore-input"
placeholder="Examples: stripe_invoice, hubspot_contacts, users"
autoComplete="off"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
/>
{({ value = '', onChange }) => (
<LemonInput
data-attr="table-name"
className="ph-ignore-input"
placeholder="Examples: stripe_invoice, hubspot_contacts, users"
autoComplete="off"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
value={value}
onChange={onChange}
/>
)}
</LemonField>
<div className="text-muted text-xs mb-4">This will be the table name used when writing queries</div>
<LemonField name="url_pattern" label="Files URL pattern">
<LemonInput
data-attr="table-name"
className="ph-ignore-input"
placeholder={ProviderMappings[provider].fileUrlPatternPlaceholder}
autoComplete="off"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
/>
{({ value = '', onChange }) => (
<LemonInput
data-attr="table-name"
className="ph-ignore-input"
phixMe marked this conversation as resolved.
Show resolved Hide resolved
placeholder={ProviderMappings[provider].fileUrlPatternPlaceholder}
autoComplete="off"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
value={value}
onChange={onChange}
/>
)}
</LemonField>
<div className="text-muted text-xs mb-4">
You can use <strong>*</strong> to select multiple files.
</div>
<LemonField name="format" label="File format" className="w-max mb-4">
<LemonSelect
data-attr="table-format"
options={[
{ label: 'Parquet (recommended)', value: 'Parquet' },
{ label: 'CSV', value: 'CSV' },
{ label: 'CSV with headers', value: 'CSVWithNames' },
{ label: 'JSON', value: 'JSONEachRow' },
{ label: 'Delta', value: 'Delta' },
]}
/>
{({ value = '', onChange }) => (
<LemonSelect
data-attr="table-format"
options={[
{ label: 'Parquet (recommended)', value: 'Parquet' },
{ label: 'CSV', value: 'CSV' },
{ label: 'CSV with headers', value: 'CSVWithNames' },
{ label: 'JSON', value: 'JSONEachRow' },
{ label: 'Delta', value: 'Delta' },
]}
value={value}
onChange={onChange}
/>
)}
</LemonField>
<LemonField name={['credential', 'access_key']} label={ProviderMappings[provider].accessKeyLabel}>
<LemonInput
data-attr="access-key"
className="ph-ignore-input"
placeholder={ProviderMappings[provider].accessKeyPlaceholder}
autoComplete="off"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
/>
{({ value = '', onChange }) => (
<LemonInput
data-attr="access-key"
className="ph-ignore-input"
placeholder={ProviderMappings[provider].accessKeyPlaceholder}
autoComplete="off"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
value={value}
onChange={onChange}
/>
)}
</LemonField>
<LemonField name={['credential', 'access_secret']} label={ProviderMappings[provider].accessSecretLabel}>
<LemonInput
data-attr="access-secret"
className="ph-ignore-input"
type="password"
placeholder="eg: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
autoComplete="off"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
/>
{({ value = '', onChange }) => (
<LemonInput
data-attr="access-secret"
className="ph-ignore-input"
type="password"
placeholder="eg: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
autoComplete="off"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
value={value}
onChange={onChange}
/>
)}
</LemonField>
{provider === 'google-cloud' && (
<div className="text-muted text-xs">
Expand All @@ -127,6 +151,13 @@ export function DatawarehouseTableForm(): JSX.Element {
</Link>
</div>
)}
{!!onUpdate && (
<div className="flex justify-end">
<LemonButton type="primary" onClick={onUpdate}>
Update
</LemonButton>
</div>
phixMe marked this conversation as resolved.
Show resolved Hide resolved
)}
</div>
</Form>
)
Expand Down
83 changes: 33 additions & 50 deletions frontend/src/scenes/data-warehouse/new/NewSourceWizard.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { PageHeader } from 'lib/components/PageHeader'
import { FEATURE_FLAGS } from 'lib/constants'
import { featureFlagLogic } from 'lib/logic/featureFlagLogic'
import { useCallback } from 'react'
import { DataWarehouseSourceIcon } from 'scenes/data-warehouse/settings/DataWarehouseSourceIcon'
import { SceneExport } from 'scenes/sceneTypes'

import { ManualLinkSourceType, SourceConfig } from '~/types'
Expand All @@ -13,7 +14,6 @@ import SchemaForm from '../external/forms/SchemaForm'
import SourceForm from '../external/forms/SourceForm'
import { SyncProgressStep } from '../external/forms/SyncProgressStep'
import { DatawarehouseTableForm } from '../new/DataWarehouseTableForm'
import { RenderDataWarehouseSourceIcon } from '../settings/DataWarehouseManagedSourcesTable'
import { dataWarehouseTableLogic } from './dataWarehouseTableLogic'
import { sourceWizardLogic } from './sourceWizardLogic'

Expand Down Expand Up @@ -52,17 +52,8 @@ interface NewSourcesWizardProps {
export function NewSourcesWizard({ onComplete }: NewSourcesWizardProps): JSX.Element {
const wizardLogic = sourceWizardLogic({ onComplete })

const {
modalTitle,
modalCaption,
isWrapped,
currentStep,
isLoading,
canGoBack,
canGoNext,
nextButtonText,
showSkipButton,
} = useValues(wizardLogic)
const { modalTitle, modalCaption, isWrapped, currentStep, isLoading, canGoBack, canGoNext, nextButtonText } =
useValues(wizardLogic)
const { onBack, onSubmit } = useActions(wizardLogic)
const { tableLoading: manualLinkIsLoading } = useValues(dataWarehouseTableLogic)

Expand Down Expand Up @@ -96,7 +87,7 @@ export function NewSourcesWizard({ onComplete }: NewSourcesWizardProps): JSX.Ele
</LemonButton>
</div>
)
}, [currentStep, isLoading, manualLinkIsLoading, canGoNext, canGoBack, nextButtonText, showSkipButton])
}, [currentStep, isLoading, manualLinkIsLoading, canGoNext, canGoBack, nextButtonText, onBack, onSubmit])

return (
<>
Expand Down Expand Up @@ -163,33 +154,29 @@ function FirstStep(): JSX.Element {
{
title: 'Source',
width: 0,
render: function RenderAppInfo(_, sourceConfig) {
return <RenderDataWarehouseSourceIcon type={sourceConfig.name} />
render: function (_, sourceConfig) {
return <DataWarehouseSourceIcon type={sourceConfig.name} />
},
},
{
title: 'Name',
key: 'name',
render: function RenderName(_, sourceConfig) {
return (
<span className="font-semibold text-sm gap-1">
{sourceConfig.label ?? sourceConfig.name}
</span>
)
},
render: (_, sourceConfig) => (
<span className="font-semibold text-sm gap-1">
{sourceConfig.label ?? sourceConfig.name}
</span>
),
},
{
key: 'actions',
width: 0,
render: function RenderActions(_, sourceConfig) {
return (
<div className="flex flex-row justify-end">
<LemonButton onClick={() => onClick(sourceConfig)} className="my-2" type="primary">
Link
</LemonButton>
</div>
)
},
render: (_, sourceConfig) => (
<div className="flex flex-row justify-end">
<LemonButton onClick={() => onClick(sourceConfig)} className="my-2" type="primary">
Link
</LemonButton>
</div>
),
},
]}
/>
Expand All @@ -208,33 +195,29 @@ function FirstStep(): JSX.Element {
{
title: 'Source',
width: 0,
render: function RenderAppInfo(_, sourceConfig) {
return <RenderDataWarehouseSourceIcon type={sourceConfig.type} />
},
render: (_, sourceConfig) => <DataWarehouseSourceIcon type={sourceConfig.type} />,
},
{
title: 'Name',
key: 'name',
render: function RenderName(_, sourceConfig) {
return <span className="font-semibold text-sm gap-1">{sourceConfig.name}</span>
},
render: (_, sourceConfig) => (
<span className="font-semibold text-sm gap-1">{sourceConfig.name}</span>
),
},
{
key: 'actions',
width: 0,
render: function RenderActions(_, sourceConfig) {
return (
<div className="flex flex-row justify-end">
<LemonButton
onClick={() => onManualLinkClick(sourceConfig.type)}
className="my-2"
type="primary"
>
Link
</LemonButton>
</div>
)
},
render: (_, sourceConfig) => (
<div className="flex flex-row justify-end">
<LemonButton
onClick={() => onManualLinkClick(sourceConfig.type)}
className="my-2"
type="primary"
>
Link
</LemonButton>
</div>
),
},
]}
/>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ export const dataWarehouseTableLogic = kea<dataWarehouseTableLogicType>([
updateTableSuccess: async ({ table }) => {
lemonToast.success(<>Table {table.name} updated</>)
actions.editingTable(false)
actions.loadDatabase()
router.actions.replace(urls.dataWarehouse())
},
})),
Expand All @@ -92,6 +93,13 @@ export const dataWarehouseTableLogic = kea<dataWarehouseTableLogicType>([
table: {
defaults: { ...NEW_WAREHOUSE_TABLE } as DataWarehouseTable,
errors: ({ name, url_pattern, credential, format }) => {
const HOGQL_TABLE_NAME_REGEX = /^[a-zA-Z_][a-zA-Z0-9_]*$/
phixMe marked this conversation as resolved.
Show resolved Hide resolved
if (!HOGQL_TABLE_NAME_REGEX.test(name)) {
return {
name: 'Invalid table name. Table names must start with a letter or underscore and contain only alphanumeric characters or underscores.',
}
}

if (url_pattern?.startsWith('s3://')) {
return {
url_pattern:
Expand Down
Loading
Loading