feat: add DataMarket — self-service data catalog on Databricks#919
Open
rautsamir wants to merge 1 commit into
Open
feat: add DataMarket — self-service data catalog on Databricks#919rautsamir wants to merge 1 commit into
rautsamir wants to merge 1 commit into
Conversation
DataMarket is a production-ready data catalog built on Databricks Apps, Unity Catalog, Lakebase, and FMAPI. It provides self-service data product discovery, governed access requests, and automated UC grant execution for enterprise and public sector customers. - One-step deploy via deploy.sh (CLI path) - Marketplace-ready via manifest.yaml build-on-startup - No external dependencies — 100% Databricks native
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces DataMarket, a Databricks Apps–hosted self-service data catalog with a React/Vite frontend and an Express + Lakebase (Postgres) backend, including governed access-request workflows and optional Unity Catalog GRANT execution via SQL Warehouse.
Changes:
- Adds a Vite + React UI with Tailwind styling, layout/navigation, and new pages (Home, Library/Manage, About/FAQ/Contact, Feature Requests, admin tabs).
- Adds Express API routes for config/settings, identity/users/groups, access requests (approve/deny/revoke), feature requests (votes/status), demo utilities, and “Ask Catalog” (FMAPI-backed search).
- Adds deployment artifacts (manifest/app yaml), package tooling, and static assets; updates CODEOWNERS for the new folder.
Reviewed changes
Copilot reviewed 65 out of 70 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| datamarket/vite.config.js | Vite build/dev configuration and path aliasing. |
| datamarket/tailwind.config.js | Tailwind theme extension (colors/typography) and plugin wiring. |
| datamarket/src/styles/responsive.css | Tailwind @apply helper classes for responsive typography and touch targets. |
| datamarket/src/pages/FeatureRequestPage.jsx | UI for creating/voting on feature/data requests. |
| datamarket/src/pages/FAQPage.jsx | Configurable FAQ page with collapsible items. |
| datamarket/src/pages/DataMarketLibraryPage.jsx | Unified “My Data / Manage” page with steward/admin tabs and sync/import actions. |
| datamarket/src/pages/DataMarketHomePage.jsx | Landing page with “Ask AI” entry, featured products, and recently accessed items. |
| datamarket/src/pages/ContactPage.jsx | Configurable contact/admin info page. |
| datamarket/src/pages/admin/AnalystMyDataView.jsx | Table view for analysts’ approved items with UC explore/copy SQL actions. |
| datamarket/src/pages/admin/AdminProductsTab.jsx | Steward product management table with inline edit and delete actions. |
| datamarket/src/pages/admin/adminConstants.js | Shared admin UI constants (colors/statuses/roles). |
| datamarket/src/pages/admin/AdminApprovalsTab.jsx | Wrapper for approvals with warning when SQL Warehouse isn’t configured. |
| datamarket/src/pages/AboutPage.jsx | “About” page with configurable text and CTA links. |
| datamarket/src/main.jsx | React root bootstrap. |
| datamarket/src/index.css | Tailwind base layer + CSS variables/theme setup. |
| datamarket/src/context/PersonaContext.jsx | Client identity/demo persona logic and request/library fetching helpers. |
| datamarket/src/context/AppConfigContext.jsx | Client-side app config fetching/defaults. |
| datamarket/src/config/navigation.js | Navigation configuration model (primary/secondary) utilities. |
| datamarket/src/components/ui/sheet.jsx | Radix-based sheet component (shadcn-style). |
| datamarket/src/components/ui/separator.jsx | Radix separator component wrapper. |
| datamarket/src/components/ui/navigation-menu.jsx | Radix navigation menu component wrapper. |
| datamarket/src/components/ui/dropdown-menu.jsx | Radix dropdown menu component wrapper. |
| datamarket/src/components/ui/dialog.jsx | Radix dialog component wrapper. |
| datamarket/src/components/ui/card.jsx | Card UI primitives. |
| datamarket/src/components/ui/button.jsx | Button primitive with variants (CVA). |
| datamarket/src/components/ui/badge.jsx | Badge primitive with variants (CVA). |
| datamarket/src/components/layout/DataMarketLayout.jsx | Top-level app layout, nav, persona/demo switcher, notifications, assistant mount. |
| datamarket/src/components/DatabricksLogo.jsx | App branding/logo component. |
| datamarket/src/components/DatabricksChart.jsx | Recharts-based chart card component. |
| datamarket/src/components/DatabricksCard.jsx | Metric card component with variants/trend display. |
| datamarket/src/App.jsx | Client-side page router + providers + first-run wizard gating. |
| datamarket/routes/users.js | Backend identity resolution + admin user/group/SCIM endpoints. |
| datamarket/routes/requests.js | Backend access request CRUD + approval/deny/revoke + notifications/library/audit. |
| datamarket/routes/insights.js | Placeholder insights routes module. |
| datamarket/routes/feature-requests.js | Backend feature request submission/voting/status APIs. |
| datamarket/routes/demo.js | Backend demo seed/reset APIs guarded by DEMO_MODE. |
| datamarket/routes/config.js | Backend health/config/settings and UC access/grant utilities. |
| datamarket/routes/ask-catalog.js | Backend FMAPI-powered semantic search over catalog metadata. |
| datamarket/README.md | App documentation, deploy notes, and architecture overview. |
| datamarket/public/datamarket-logo.svg | DataMarket logo asset. |
| datamarket/public/databricks-logo.svg | Databricks logo asset. |
| datamarket/public/databricks-favicon.svg | Favicon asset. |
| datamarket/postcss.config.js | PostCSS pipeline configuration for Tailwind. |
| datamarket/package.json | App dependencies/scripts (Vite + Express + Tailwind). |
| datamarket/manifest.yaml | Marketplace manifest/entrypoint and required resources/envs. |
| datamarket/jsconfig.json | Path aliasing for editor tooling (@/*). |
| datamarket/index.html | Vite HTML entrypoint and font preconnects. |
| datamarket/db.js | Lakebase connection/pool management and runtime migrations/seed logic. |
| datamarket/databricks.js | Databricks API helpers (SQL execution, UC schema fetch, RFA notify). |
| datamarket/components.json | shadcn/ui component generator config. |
| datamarket/auth.js | OAuth/token helpers + Lakebase credential resolution. |
| datamarket/app.yaml.example | Example Databricks Apps YAML for CLI-based deployment. |
| datamarket/app.yaml | Databricks Apps YAML (currently env-specific values). |
| datamarket/app.js | Express server bootstrap, middleware, route registration, SPA fallback. |
| CODEOWNERS | Adds code ownership entry for the new datamarket folder. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1
to
+6
| /** @type {import('tailwindcss').Config} */ | ||
| export default { | ||
| darkMode: ["class"], | ||
| content: [ | ||
| "./index.html", | ||
| "./src/**/*.{js,ts,jsx,tsx}", |
Comment on lines
+3
to
+30
| .text-responsive-display-1 { | ||
| @apply text-display-1 lg:text-display-1-lg; | ||
| } | ||
|
|
||
| .text-responsive-display-2 { | ||
| @apply text-display-2 lg:text-display-2-lg; | ||
| } | ||
|
|
||
| .text-responsive-h1 { | ||
| @apply text-h1 lg:text-h1-lg; | ||
| } | ||
|
|
||
| .text-responsive-h2 { | ||
| @apply text-h2 lg:text-h2-lg; | ||
| } | ||
|
|
||
| /* Touch-friendly button classes */ | ||
| .btn-touch { | ||
| @apply min-h-touch min-w-touch flex items-center justify-center; | ||
| } | ||
|
|
||
| .btn-touch-sm { | ||
| @apply min-h-touch-sm min-w-touch-sm flex items-center justify-center; | ||
| } | ||
|
|
||
| .btn-touch-lg { | ||
| @apply min-h-touch-lg min-w-touch-lg flex items-center justify-center; | ||
| } |
Comment on lines
+102
to
+112
| @layer base { | ||
| :root { | ||
| --background: 0 0% 100%; | ||
| --foreground: 0 0% 3.9%; | ||
| --card: 0 0% 100%; | ||
| --card-foreground: 0 0% 3.9%; | ||
| --popover: 0 0% 100%; | ||
| --popover-foreground: 0 0% 3.9%; | ||
| --primary: 0 0% 9%; | ||
| --primary-foreground: 0 0% 98%; | ||
| --secondary: 0 0% 96.1%; |
Comment on lines
+265
to
+268
| const hasApprovedAccess = useCallback((productRef) => { | ||
| const refStr = toProductRef(productRef) | ||
| if (persona.approvedProductRefs === 'all') return false | ||
| if (persona.approvedProductRefs.includes(refStr)) return true |
Comment on lines
+1
to
+28
| command: | ||
| - "node" | ||
| - "app.js" | ||
| env: | ||
| # ── Databricks Identity ──────────────────────────────────────────────────── | ||
| - name: DATABRICKS_HOST | ||
| value: "https://dbc-7e8bd036-20cc.cloud.databricks.com" | ||
|
|
||
| # ── Admin bootstrap ──────────────────────────────────────────────────────── | ||
| # On first SSO login the app auto-promotes this email to admin. | ||
| # Set this to your own admin email before deploying. | ||
| - name: ADMIN_EMAIL | ||
| value: "admin@example.com" | ||
|
|
||
| # ── Lakebase Connection ──────────────────────────────────────────────────── | ||
| - name: LAKEBASE_HOST | ||
| value: "ep-spring-dust-d8wvin0s.database.us-east-2.cloud.databricks.com" | ||
| - name: LAKEBASE_DB | ||
| value: "databricks_postgres" | ||
| - name: LAKEBASE_SCHEMA | ||
| value: "datamarket" | ||
| - name: LAKEBASE_ENDPOINT | ||
| value: "projects/datamarket/branches/production/endpoints/primary" | ||
|
|
||
| # ── Mode ────────────────────────────────────────────────────────────────── | ||
| # false = real SSO identity + UC grants. true = persona switcher (demos only) | ||
| - name: DEMO_MODE | ||
| value: "false" |
Comment on lines
+76
to
+103
| // ─── Portal Settings (admin CRUD) ──────────────────────────────────────────── | ||
| app.get('/api/portal/settings', async (req, res) => { | ||
| try { | ||
| const s = await loadSettings(); | ||
| res.json(s); | ||
| } catch (e) { | ||
| res.status(500).json({ error: e.message }); | ||
| } | ||
| }); | ||
|
|
||
| app.put('/api/portal/settings', async (req, res) => { | ||
| try { | ||
| const updates = req.body; // { key: value, ... } | ||
| if (!updates || typeof updates !== 'object') return res.status(400).json({ error: 'Body must be a JSON object of {key: value}' }); | ||
| for (const [key, value] of Object.entries(updates)) { | ||
| await query( | ||
| `INSERT INTO settings (key, value, updated_at) VALUES ($1, $2, NOW()) | ||
| ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value, updated_at = NOW()`, | ||
| [key, value ?? ''] | ||
| ); | ||
| } | ||
| invalidateSettingsCache(); | ||
| await loadSettings(); | ||
| res.json({ ok: true }); | ||
| } catch (e) { | ||
| res.status(500).json({ error: e.message }); | ||
| } | ||
| }); |
Comment on lines
+92
to
+118
| app.put('/api/portal/requests/:id/approve', async (req, res) => { | ||
| try { | ||
| const { id } = req.params; | ||
| const { adminEmail = 'datasteward@example.org' } = req.body; | ||
|
|
||
| // id can be a UUID or a request_ref like "REQ-001" — avoid type mismatch | ||
| const isUUID = /^[0-9a-f-]{36}$/i.test(id); | ||
| const { rows: [req_] } = await query(` | ||
| SELECT ar.*, dp.uc_full_name, dp.display_name AS product_name, u.email AS requester_email | ||
| FROM access_requests ar | ||
| JOIN data_products dp ON dp.product_id = ar.product_id | ||
| JOIN users u ON u.user_id = ar.requester_id | ||
| WHERE ${isUUID ? 'ar.request_id = $1::uuid' : 'ar.request_ref = $1'}`, [id]); | ||
| if (!req_) return res.status(404).json({ error: 'Request not found' }); | ||
|
|
||
| const ucGrantSql = req_.uc_full_name | ||
| ? `GRANT SELECT ON ${req_.uc_full_name} TO \`${req_.requester_email}\`;` | ||
| : `-- No UC table linked for ${req_.product_name} (set uc_full_name on the product to enable automatic grants)`; | ||
|
|
||
| const { rows: [adminUser] } = await query('SELECT user_id FROM users WHERE email = $1', [adminEmail]); | ||
|
|
||
| await query(`UPDATE access_requests SET status = 'Approved', resolved_at = NOW(), | ||
| resolved_by = $1, uc_grant_issued = TRUE, uc_grant_sql = $2, updated_at = NOW(), | ||
| expires_at = NOW() + INTERVAL '90 days' | ||
| WHERE request_ref = $3`, | ||
| [adminUser?.user_id || null, ucGrantSql, req_.request_ref]); | ||
|
|
Comment on lines
+95
to
+124
| // ─── Admin: Users ───────────────────────────────────────────────────────────── | ||
| app.get('/api/portal/admin/users', async (req, res) => { | ||
| try { | ||
| const { rows } = await query( | ||
| `SELECT user_id, email, display_name, role, department, created_at FROM users ORDER BY display_name`); | ||
| res.json(rows.map(u => ({ ...u, role: normalizeRole(u.role) }))); | ||
| } catch (e) { | ||
| res.status(500).json({ error: e.message }); | ||
| } | ||
| }); | ||
|
|
||
| app.post('/api/portal/admin/users', async (req, res) => { | ||
| try { | ||
| const { email, display_name, role, department } = req.body; | ||
| if (!email) return res.status(400).json({ error: 'email is required' }); | ||
| const normalizedRole = normalizeRole(role); | ||
| const { rows: [user] } = await query( | ||
| `INSERT INTO users (email, display_name, role, department) | ||
| VALUES ($1, $2, $3, $4) | ||
| ON CONFLICT (email) DO UPDATE | ||
| SET display_name = EXCLUDED.display_name, | ||
| role = EXCLUDED.role, | ||
| department = EXCLUDED.department | ||
| RETURNING *`, | ||
| [email.trim().toLowerCase(), display_name || '', normalizedRole, department || ''] | ||
| ); | ||
| res.status(201).json({ ...user, role: normalizeRole(user.role) }); | ||
| } catch (e) { | ||
| res.status(500).json({ error: e.message }); | ||
| } |
Comment on lines
+68
to
+70
| const { rows: [{ count }] } = await query('SELECT COUNT(*) FROM access_requests', []); | ||
| const ref = `REQ-${String(parseInt(count) + 1).padStart(3, '0')}`; | ||
|
|
Comment on lines
+1
to
+25
| import { defineConfig } from 'vite' | ||
| import react from '@vitejs/plugin-react' | ||
| import path from 'path' | ||
|
|
||
| export default defineConfig({ | ||
| plugins: [react()], | ||
| build: { | ||
| outDir: 'dist', | ||
| assetsDir: 'assets', | ||
| }, | ||
| server: { | ||
| port: 3001, | ||
| proxy: { | ||
| '/api': { | ||
| target: 'http://localhost:3000', | ||
| changeOrigin: true, | ||
| }, | ||
| }, | ||
| }, | ||
| resolve: { | ||
| alias: { | ||
| '@': path.resolve(__dirname, './src'), | ||
| }, | ||
| }, | ||
| }) No newline at end of file |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds DataMarket, a self-service data catalog and governed access management app built entirely on Databricks.
What it does
GRANTstatements via SQL WarehouseDatabricks services
Databricks Apps · Unity Catalog · Lakebase Autoscaling (managed Postgres) · Foundation Model APIs · SQL Statement Execution API
Deployment
./deploy.sh --profile my-profilemanifest.yamlbuilds frontend on first startup (no pre-build needed)Links
Checklist
datamarket/README.mdwith required frontmatter (title, language, author, date, tags)@rautsamirtoCODEOWNERSin alphabetical orderdist/andnode_modules/excluded