Databricks
One SQL statement for a self-refreshing reporting table in Databricks
Create a Databricks SQL materialized view that refreshes daily, documents its columns, and calculates reporting metrics in a single statement.
Loading
Delta Lake Schema Evolution: How to Add Columns Without Breaking Streaming Jobs
Learn how to add columns to Delta Lake tables with schema evolution, use mergeSchema in PySpark, and safely manage compatibility with streaming jobs.
Databricks Lakeflow Pipelines Now Support Parameters
Learn how to define, reference, and override Databricks Lakeflow Pipeline parameters to reuse SQL pipelines across environments, sources, and date ranges.
Databricks Apps Can Use Workspace-Level Python and npm Repositories
Learn how Databricks Apps uses workspace-level Python and npm repositories, letting administrators manage package sources while developers declare app dependencies.
Unity Catalog Adds Governed Tag Propagation to Derived Tables and Views
Learn how Databricks Unity Catalog propagates governed tags to derived tables and views, how it supports ABAC, and why asynchronous propagation affects policy enforcement.
Databricks Opens the Unity Gateway Skills API and SDKs in Beta
Learn how Databricks Unity Gateway Skills API and SDKs let teams create, publish, govern, and share reusable AI agent instruction bundles through Unity Catalog.
Databricks Can Repartition Stateful Streaming Queries Without Losing Checkpoints
Learn how Databricks on-demand state repartitioning changes stateful Spark Structured Streaming partition counts while preserving an existing checkpoint.
💡 NEW: Create Relationships in Databricks AI/BI Dashboard
Learn how Databricks AI/BI Dashboard dataset relationships connect data sources, enable cross-filtering, and support more interactive self-service analytics.
Stop Hiding Your Credentials in Notebooks: A Look at Secrets in Unity Catalog
Learn how Databricks Unity Catalog secrets centralize credential storage, permissions, rotation, auditing, and secure use in notebooks and Python UDFs.
Databricks Auto Loader, Properly Explained: The Options Nobody Bothers to Justify
In this article, you'll learn how Databricks Auto Loader actually works under the hood - not just the five-line tutorial version. It covers file discovery (directory listing vs. event-based), the five schema evolution modes and when to use each, throughput controls like maxFilesPerTrigger and maxBytesPerTrigger, file metadata and lineage, the built-in cloud_files_state() audit function, trigger strategy, and what Auto Loader's exactly-once guarantee actually promises (and doesn't
Reading and Writing JSON Files in Databricks Unity Catalog
In this article, I covered how to read and write JSON multiline data from Databricks' unity catalog volume to delta table
DATABRICKS.SQL and DATABRICKS.TABLE Functions Debuts in Microsoft Excel
Discover how Excel’s DATABRICKS.SQL and DATABRICKS.TABLE functions connect spreadsheets directly to Databricks, enabling live SQL queries, enterprise data analysis, and real-time insights.
Understanding EXPLAIN FORMATTED in Databricks SQL
Master Databricks SQL query optimization with EXPLAIN FORMATTED. Uncover Spark's execution plans, identify bottlenecks, and tune performance.
Why Every Databricks Data Engineer Should Audit Their Query History
Audit Databricks query history to identify and optimize costly, long-running queries, ensuring financial governance and platform performance.
Why Databricks Catalog Commits is a Game-Changer for Lakehouse Governance and Transactions
Databricks Catalog Commits revolutionize Lakehouse governance with multi-table ACID transactions, enhanced security, and improved performance.
Stripe + Databricks: Finally, Real-Time Payments Data Without the Headache
Unlock real-time Stripe payment data in Databricks without ETL! Access transactions, customers, and more instantly for faster analytics and AI. No pipelines needed!
Building a reusable access management layer on Databricks with Python, Notebooks and Genie spaces
What if access requests in Databricks could be handled like a real product instead of a manual admin task? In this article, I show how I built a reusable access management with Python, Git backed Databricks repos, notebooks and Genie Spaces to turn business friendly access requests into governed Unity Catalog permissions. The result is a cleaner, auditable and more scalable way to manage access while also validating the end user experience through natural language data exploration.
Disable Tasks in Databricks Lakeflow Jobs: A Powerful Feature for Flexible Workflow Orchestration
Simplify Databricks Lakeflow Jobs with disabled tasks! Boost flexibility, streamline debugging, and optimize deployments. A must-have for efficient data pipelines.
How Genie Code is Transforming Data Workflows in Databricks
Discover how Databricks Genie Code transforms data workflows from AI-assisted coding to autonomous agents. Explore its capabilities and impact on data teams.
Why Custom URLs in Azure Databricks Are a Game-Changer for Enterprise Teams
Simplify Azure Databricks access with custom URLs! Learn how this enterprise feature creates a unified, branded entry point, improving user experience and governance. Say goodbye to juggling multiple workspace URLs!
Databricks Runtime 18.1 (Beta) — A Big Leap Forward for Data & AI Teams
Databricks Runtime 18.1 (Beta) delivers significant enhancements for data and AI teams! Expect faster pipelines, smarter features, and improved reliability.
What Is the Role of Databricks in Big Data?
Discover Databricks' pivotal role in big data: a unified platform for data engineering, real-time analytics, and machine learning. Explore its cloud-native architecture.
Lakehouse vs Data Warehouse: Enterprise Reality Check
Explore Lakehouse vs. Data Warehouse for enterprise analytics. Understand architectural differences, governance, scalability, and when each approach excels. Hybrid strategies revealed.
UNDROP in Azure Databricks: Recover Dropped Tables Like a Pro
In this article, I covered how to use the UNDROP command in Azure Databricks with Unity Catalog to recover accidentally dropped tables and materialized views. A data engineer's lifesaver! Discover syntax, permissions, and limitations for effective data recovery.
How to Import a Notebook into Your Databricks Workspace (Step-by-Step Guide)
Learn how to import notebooks into your Databricks workspace with this step-by-step guide. Master drag-and-drop, URL imports, and more. Start building data workflows quickly!
How to Load Data from Snowflake to Databricks: Two Practical Methods
Learn two ways to move data from Snowflake to Databricks: manual export for small or one-time transfers, and real-time sync via managed pipelines. This article covers methods, tools, and best practices for efficient, scalable, low-latency data transfer between warehouse and lakehouse.
On-Heap vs Off-Heap Memory Management in Databricks
Learn how on-heap and off-heap memory work in Databricks and Apache Spark, including their trade-offs, configuration options, and performance considerations.
Databricks Genie: Unlocking AI-Powered Data Insights with OpenAI
Databricks Genie is a conversational AI assistant powered by Azure OpenAI to enable self-service analytics, and conduct data exploration using natural language questions to produce data-driven insights.
Databricks Unity Catalog: Mastering Metastore and Namespace
Databricks Unity Catalog provides a unified governance layer with a three-level namespace (Catalog, Schema, Table) for efficient data organization, access control, and sharing across workspaces, improving security and collaboration.
Understanding the AI_EXTRACT SQL Function in Databricks
Learn how to use Databricks AI_EXTRACT to pull entities such as people, dates, locations, and organizations from unstructured text with SQL.
