Platform
- Databricks Runtime 11.2 and 11.2 ML are GA
- Support for AWS Graviton instances is GA
- You can use Data Explorer to search for tables that are registered in Unity Catalog, only the tables that you have permissions to see appear in search results
- View and organize assets in the workspace browser across personas
- Table schemas now support default values for columns, this behavior is available for csv, parquet,json and orc. => Example : Create table t (id Int, data int default 42) (DBR 11.2)
- Any_value agg function return any random values of expr for a group of rows (DBR 11.2)
- Delta Cache is now Disk Cache. For more information : Link
- Audit logs now include events for managing credentials for git repos.
- Audit logs now include events related to the web terminal. For more information : Link
Governance
- Dbutils.fs commands are now allowed on user isolation clusters with UC
- Create catalog, create external location, create share, create provider privileges can now be granted on Unity catalog metastores
- Alter share now support start version
- Data Lineage is now Public Preview
- Unity Catalog is now GA in the us-west-1
Delta Lake
- Convert to delta automatically infers the partition schema for parquet table registered to hive metastore or Unity Catalog eliminating the need to provide partitioned clause (DBR 11.2)
Databricks SQL
- Include Databricks SQL tasks in Databricks Workflows, including running queries, refreshing dashboards, alerts. It’s required to have serverless SQL warehouses activated.
- Sql Warehouse Endpoint ( Current mode)
-Describe and show SQL UDFs now show Unity Catalog names in their output.
-Objects created in Unity Catalog have information schema support.
-A Single Metastore per available region is supported.
- A Metastore can have up to 1,000 catalogs.
- A Catalog can have up to 10,000 schemas.
- A schema can have up to 10,000 tables.
-SQL UDFs are now in public preview in Unity Catalog.
-SQL syntax for external locations in Unity Catalog:
- CREATE |DROP | ALTER| DESCRIBE SHOW EXTERNAL LOCATION is fully supported.
- GRANT/REVOKE/SHOW permissions for principals on specific EXTERNAL LOCATION is fully supported
Delta Live Tables
- Select clusters policies directly in the Delta Live Tables UI
Security
- Compliance security profile workspaces support i4i instance types.For more information: Link
- Customer Managed keys for encryption which includes both managed services and workspace storage is GA. For more information : Link
Machine Learning
- AutoML has a better support for imbalance Datasets for classification problems ( DBR 11.2)