Databricks SME

Key Responsibilities: Architecture & Platform Design Design enterprise Databricks Lakehouse architectures aligned with the Databricks Well-Architected Framework Define reference architectures for batch, streaming, analytics, and ML workloads Select and standardize cluster, compute, and workspace architectures Design multi-workspace strategies (dev/test/prod, shared vs. isolated) Ensure architectures meet scalability, availability, and performance requirements Well-Architected Framework Alignment Apply Databricks best practices across all pillars, including: Security & Governance (Unity Catalog, IAM, data access controls) Reliability & Resilience (job retries, checkpointing, failure isolation) Performance Efficiency (cluster sizing, autoscaling, caching) Cost Optimization (compute policies, workload separation, monitoring) Operational Excellence (monitoring, automation, CI/CD, runbooks) Implementation & Engineering Lead Databricks workspace, cluster, and Unity Catalog implementations Implement Delta Lake, Delta Live Tables (DLT), and Structured Streaming Build and optimize ETL/ELT pipelines using Spark and SQL Integrate Databricks with cloud services (S3/ADLS/GCS, IAM, Key Vault, networking) Establish CI/CD pipelines for notebooks, jobs, and infrastructure Security, Governance & Compliance Implement Unity Catalog for centralized governance Define data classification, lineage, and audit strategies Enforce least-privilege access and secure networking patterns