What’s new in Databricks for September 2022

Platform 

  • Databricks Runtime 11.2 and 11.2 ML are GA
  • Support for AWS Graviton instances is GA
  • You can use Data Explorer to search for tables  that are registered in Unity Catalog, only the tables that you have permissions to see appear in search results
  • View and organize assets in the workspace browser across personas
  • Table schemas now support default values for columns, this behavior is available for csv, parquet,json and orc. => Example : Create table t (id Int, data int default 42)  (DBR 11.2)
  • Any_value agg function return any random values of expr for a group of rows (DBR 11.2)
  • Delta Cache is now Disk Cache. For more information : Link
  • Audit logs now include events for managing credentials for git repos. 
  • Audit logs now include events related to the web terminal. For more information : Link

Governance

  • Dbutils.fs commands are now allowed on user isolation clusters with UC
  • Create catalog, create external location, create share, create provider privileges can now be granted on Unity catalog metastores
  • Alter share now support start version
  • Data Lineage is now Public Preview
  • Unity Catalog is now GA in the us-west-1

Delta Lake

  • Convert to delta automatically infers the partition schema for parquet table registered to hive metastore or Unity Catalog eliminating the need to provide partitioned clause (DBR 11.2)

Databricks SQL

  • Include Databricks SQL tasks in Databricks Workflows, including running queries, refreshing dashboards, alerts. It’s required to have serverless SQL warehouses activated.
  • Sql Warehouse Endpoint ( Current mode)

-Describe and show SQL UDFs now show Unity Catalog names in their output.

-Objects created in Unity Catalog have information schema support.

-A Single Metastore per available region is supported.

  • A Metastore can have up to 1,000 catalogs.
  • A Catalog can have up to 10,000 schemas.
  • A schema can have up to 10,000 tables.

-SQL UDFs are now in public preview in Unity Catalog.

-SQL syntax for external locations in Unity Catalog:

  • CREATE |DROP | ALTER| DESCRIBE SHOW EXTERNAL LOCATION is fully supported.
  • GRANT/REVOKE/SHOW permissions for principals on specific EXTERNAL LOCATION is fully supported

Delta Live Tables

  • Select clusters policies directly in the Delta Live Tables UI

Security 

  • Compliance security profile workspaces support i4i instance types.For more information: Link
  • Customer Managed keys for encryption which includes both managed services and workspace storage is GA. For more information : Link

Machine Learning

  • AutoML has a better support for imbalance Datasets for classification problems ( DBR 11.2)

Laisser un commentaire

Votre adresse e-mail ne sera pas publiée. Les champs obligatoires sont indiqués avec *