Databricks interview questions have changed a lot as the platform has matured. Interviewers for data engineer, analytics engineer and platform roles now expect you to talk confidently about Unity Catalog governance, Lakeflow pipelines, liquid clustering and automated table maintenance, not just Spark basics and notebooks. Answers that still lean on older names and habits can make a strong candidate sound out of date.
This guide collects 30 questions with short, interview-ready answers, grouped from beginner to scenario-based, and checked against the current Databricks documentation. Use it to rehearse out loud, then go deeper in the areas your target role cares about most. If you would rather practice with a trainer and real workspaces, our live Databricks training covers the same ground with hands-on labs.
Key takeaways
- Delta Lake is the default table format on Databricks, so most technical rounds start with ACID transactions, the transaction log, time travel and MERGE.
- Unity Catalog uses a three-level namespace (catalog.schema.object) and is enabled automatically for workspaces created after November 8, 2023.
- Databricks now recommends liquid clustering for new tables instead of partitioning and ZORDER.
- Lakeflow pipelines handle declarative ETL with expectations that warn, drop or fail on bad records.
- Scenario questions test judgment: incremental ingestion, schema drift, slow queries and safe recovery after a bad write.
Beginner Databricks interview questions
1. What is Databricks?
Databricks is a cloud data and AI platform built around the lakehouse idea: one copy of data in open formats on cloud object storage, used for data engineering, SQL analytics, streaming and machine learning. It runs on AWS, Microsoft Azure and Google Cloud and is built on Apache Spark, with its own engine optimizations on top.
2. What is a lakehouse, and how is it different from a data warehouse and a data lake?
A data lake stores raw files cheaply but offers weak reliability and governance. A warehouse offers strong SQL performance and transactions but usually on proprietary storage. A lakehouse adds warehouse features such as ACID transactions, schema enforcement and fine-grained access control directly on top of open files in the lake, so BI and ML workloads share the same tables.
3. What is Delta Lake?
Delta Lake is the open source storage layer behind tables on Databricks. It stores data as Parquet files and adds a file-based transaction log (the _delta_log directory) that provides ACID transactions and scalable metadata. Unless you specify otherwise, every table you create on Databricks is a Delta table.
4. Why does the transaction log matter?
Every write produces a new table version recorded in the log. Readers see a consistent snapshot, concurrent writers are coordinated, and the history makes auditing, time travel and rollback possible. Databricks advises against editing data files or log files directly, because that can corrupt the table.
5. What is the medallion architecture?
It is a design pattern that organizes data into layers of increasing quality. Bronze holds raw ingested data, silver holds cleaned and validated data, and gold holds enriched, business-ready tables for reporting and ML. It is a guideline, not a hard rule, and teams adapt the number of layers to their needs.
6. What is Unity Catalog?
Unity Catalog is the governance layer for data and AI assets on Databricks. It handles access control, lineage, auditing, discovery and data quality monitoring. Tables, views, volumes, functions and models live in a three-level namespace, catalog.schema.object, and permissions are granted on these securable objects to users, groups or service principals.
7. What is the difference between managed and external tables?
For a managed table, Unity Catalog governs the table and also manages the lifecycle of its underlying files. For an external table, Unity Catalog governs access but the files live in a storage location you manage, so dropping the table does not remove that data. Managed tables also qualify for features such as predictive optimization.
8. What is Photon?
Photon is the Databricks native vectorized query engine. It processes data in columnar batches and speeds up SQL, DataFrame operations, ETL and stateless streaming, while staying compatible with Spark APIs, so existing code does not need to change.
Intermediate Databricks interview questions
9. How does time travel work?
Because each write creates a version, you can query an earlier state by version or timestamp:
SELECT * FROM sales.orders VERSION AS OF 42;
SELECT * FROM sales.orders TIMESTAMP AS OF '2026-11-01T00:00:00Z';
DESCRIBE HISTORY sales.orders LIMIT 5; Time travel depends on retained data files. Databricks recommends relying on only the last 7 days unless you raise both the data and log retention settings, and it warns against treating table history as a long-term backup.
10. What is the difference between log retention and deleted file retention?
Table history is kept according to logRetentionDuration, which defaults to 30 days. Time travel also needs the old data files, which are governed by deletedFileRetentionDuration, which defaults to 7 days. Mixing these two up is a common interview slip.
11. How do you upsert data into a Delta table?
Use MERGE INTO, which matches source rows to target rows on a key and updates, inserts or deletes in one atomic operation:
MERGE INTO crm.customers AS t
USING staging.customer_updates AS s
ON t.customer_id = s.customer_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *; On Databricks SQL and Databricks Runtime 12.2 LTS and above, WHEN NOT MATCHED BY SOURCE lets you update or delete target rows that no longer exist in the source, which is useful for full syncs.
12. What does VACUUM do, and what is the risk?
VACUUM removes data files that are no longer referenced by the current table state and are older than the retention threshold, which defaults to 7 days. After vacuuming, you cannot time travel to versions older than that window. Setting a very short retention is risky because long-running jobs may still be writing uncommitted files. Use VACUUM table DRY RUN to preview what would be deleted.
13. What is the difference between VACUUM FULL and VACUUM LITE?
On Databricks Runtime 16.1 and above, FULL (the default) scans the table directory and removes all unreferenced files outside the retention window. LITE uses the transaction log to find files to remove instead of listing the whole directory, which is faster on large tables, but it fails if the needed log history has already been pruned.
14. What does OPTIMIZE do?
OPTIMIZE compacts many small files into fewer, larger ones, which reduces file listing and scan overhead. On a table without liquid clustering it performs bin-packing, which is idempotent. It can also apply ZORDER BY on older layouts, and on clustered tables it incrementally clusters data.
15. What is liquid clustering, and why does it replace partitioning and ZORDER?
Liquid clustering organizes data by clustering keys you set with CLUSTER BY. Unlike partitions, you can change the keys later without rewriting existing data, and it handles skew and high-cardinality filter columns well. It is generally available for Delta tables from Databricks Runtime 15.4 LTS, and Databricks recommends it for all new tables. It cannot be combined with partitioning or ZORDER on the same table.
CREATE TABLE events.page_views (
event_date DATE, user_id BIGINT, url STRING
) CLUSTER BY (event_date, user_id);
ALTER TABLE events.page_views CLUSTER BY (user_id); 16. What is predictive optimization?
For Unity Catalog managed tables, predictive optimization automatically runs OPTIMIZE, VACUUM and ANALYZE when Databricks decides they are worthwhile, so teams spend less time scheduling maintenance by hand.
17. What are deletion vectors?
Deletion vectors let deletes, updates and merges mark rows as removed without immediately rewriting whole Parquet files. The physical rewrite happens later, for example during compaction. This makes small row-level changes much cheaper on large tables.
18. What is Auto Loader?
Auto Loader is a Structured Streaming source named cloudFiles that picks up new files from cloud storage (S3, ADLS, GCS or Unity Catalog volumes) as they arrive. It records discovered files in its checkpoint so each file is processed exactly once, even after a restart.
(spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", "/Volumes/raw/meta/orders_schema")
.load("/Volumes/raw/landing/orders/")
.writeStream
.option("checkpointLocation", "/Volumes/raw/meta/orders_ckpt")
.trigger(availableNow=True)
.toTable("bronze.orders")) 19. What are Lakeflow pipelines?
Lakeflow pipelines (previously known as Delta Live Tables and later Lakeflow Declarative Pipelines) let you declare streaming tables, materialized views and their dependencies in SQL or Python. Databricks then manages orchestration, retries, checkpoints and infrastructure. In Python the module is imported with from pyspark import pipelines as dp.
20. How do expectations work in a pipeline?
Expectations are named SQL Boolean constraints attached to a dataset. Each one has an action: warn (the default) keeps invalid rows but records metrics, drop removes them, and fail stops the update.
@dp.table
@dp.expect("valid_amount", "amount >= 0")
@dp.expect_or_drop("has_order_id", "order_id IS NOT NULL")
@dp.expect_or_fail("known_currency", "currency IN ('USD','GBP','EUR')")
def silver_orders():
return spark.readStream.table("bronze.orders") Advanced Databricks interview questions
21. What are Lakeflow Jobs?
Lakeflow Jobs (formerly Databricks Workflows) is the orchestration service in Databricks for running multiple tasks as one workflow, such as notebooks, pipelines, SQL and Python scripts, with dependencies, schedules, triggers and retries. Expect to explain how you would chain ingestion, transformation and a dashboard refresh.
22. How do you implement CI/CD on Databricks?
Use Declarative Automation Bundles (formerly Databricks Asset Bundles). A bundle keeps code and resource definitions, such as jobs and pipelines, in source control with a databricks.yml file, and the Databricks CLI validates and deploys it to targets like dev, staging and prod. Pair it with Git folders, pull request reviews and automated tests.
23. What is change data feed?
Change data feed records row-level changes between table versions so downstream jobs can process only inserts, updates and deletes instead of rereading full tables. You read it with the table_changes() function in SQL or the readChangeFeed option in Spark.
24. How does Auto Loader handle schema changes?
With a schema location set, Auto Loader infers the schema and tracks it. In the default addNewColumns mode, a new column makes the stream stop with an UnknownFieldException after the schema is updated, and a restart continues with the new schema. Values that do not fit the schema land in the _rescued_data column instead of being silently lost. Running the stream under a job with automatic retries makes this smooth.
25. How would you restore a table after a bad write?
Check DESCRIBE HISTORY to find the last good version, inspect it with time travel, then run RESTORE TABLE name TO VERSION AS OF n. The restore is itself recorded as a new version, so it can be audited. It only works while the older data files are still within retention.
26. How do you secure sensitive columns for different teams?
Grant privileges at the catalog, schema or table level in Unity Catalog, then apply row filters and column masks where people need the same table with different visibility. Attribute-based policies and tags help scale this across many tables. The audit log system table shows who accessed what.
Scenario-based Databricks interview questions
27. A dashboard query on a large table has become slow. What do you check?
Start with the query profile to see where time goes. Then check for many small files (compaction), whether the filter columns match the clustering keys, whether statistics are current, and whether Photon and a right-sized SQL warehouse are in use. For new or redesigned tables, moving to liquid clustering on the main filter columns is often the cleanest fix.
28. Files land in S3 every few minutes and the business wants data within the hour. How would you build it?
Use Auto Loader inside a Lakeflow pipeline to land raw data in a bronze streaming table, apply expectations while building silver, and produce gold materialized views. Schedule it with Lakeflow Jobs, or run it continuously if the latency target tightens. This design is incremental, restartable and easy to monitor.
29. A teammate wants to set VACUUM retention to zero hours to save storage. How do you respond?
Explain that this can delete files still needed by running jobs and removes the ability to time travel or restore. Databricks strongly recommends at least 7 days and includes a safety check for exactly this reason. Suggest predictive optimization or a normal schedule instead.
30. A partitioned table has thousands of tiny partitions. What would you do?
Over-partitioning on a high-cardinality column creates small files and slow listing. Recreate the table with liquid clustering on the columns queries actually filter on, for example with CREATE TABLE new CLUSTER BY (...) AS SELECT * FROM old, validate results, then switch consumers over. Explain that clustering keys can evolve later without a full rewrite.
How to prepare for a Databricks interview
- Practice in a real workspace: create a Delta table, run MERGE, check history, restore, and enable clustering.
- Build one small bronze, silver and gold pipeline with expectations so you can describe it end to end.
- Learn the current product names. Using Lakeflow pipelines, Lakeflow Jobs and bundles correctly shows you keep up with the platform.
- Prepare two stories from your own work about performance tuning and data quality, with honest numbers you can defend.
- Strengthen Spark and SQL fundamentals. Our data science tutorial and Snowflake interview questions are useful if your role also touches analytics or a second warehouse platform.
- Update your resume with specific Databricks features you have used. The free Tutorials Mania Resume Builder makes this quick.
- Compare learning paths for related skills such as Spark, Azure Databricks and Snowflake in our full list of IT training courses.
Learn Databricks with live, instructor-led training
Our Databricks course covers the lakehouse end to end: Apache Spark, Delta Lake, Unity Catalog, workflows, Databricks SQL and AI features. You learn in live, instructor-led classes with hands-on labs and certification guidance for the data engineer exams.
Frequently asked questions
Is Databricks hard to learn for an interview?
If you already know SQL and some Python, the core ideas come quickly. Most of the effort goes into Delta Lake behavior, Unity Catalog permissions and pipeline design, which are easier to learn by building a small project than by reading alone.
Do I need to know PySpark for a Databricks interview?
For data engineering roles, yes. Expect to write DataFrame transformations, joins, window functions and simple streaming code. Analyst roles lean more on Databricks SQL, but basic PySpark still helps.
What is the difference between Databricks and Snowflake in interviews?
Databricks questions focus on Spark, open table formats, streaming and ML workloads, while Snowflake questions focus on its warehouse architecture, virtual warehouses and SQL features. Many teams use both, so be ready to compare them fairly.
Are Delta Live Tables questions still asked?
Yes, but under newer names. Delta Live Tables is now Lakeflow pipelines. If an interviewer uses the old name, answer the question and mention the current name briefly.
Does a Databricks certification help in interviews?
A certification shows structured study and gives you a clear syllabus to work through, but interviewers still test practical skill. Treat it as a complement to project experience, not a substitute.
Sources
- Databricks documentation: What is Delta Lake in Databricks? (docs.databricks.com/aws/en/delta/)
- Databricks documentation: What is Unity Catalog?
- Databricks documentation: Work with table history (time travel, retention, RESTORE)
- Databricks documentation: VACUUM and OPTIMIZE SQL reference
- Databricks documentation: Use liquid clustering for tables
- Databricks documentation: Predictive optimization for Unity Catalog managed tables; Deletion vectors
- Databricks documentation: What is Auto Loader?; Configure schema inference and evolution in Auto Loader
- Databricks documentation: Manage data quality with pipeline expectations; Develop pipeline code with Python
- Databricks documentation: Lakeflow Jobs; What are Declarative Automation Bundles?; Change data feed; Photon; Medallion architecture; Upsert using merge
- Databricks documentation: Product name changes; What happened to Delta Live Tables (DLT)?