Without the Right Data, AI Cannot Flourish<\/strong><\/h4>\nThe reality and challenge facing D&A professionals today is that building a comprehensive and insightful analytic model depends on access to complete and quality data, in great quantity. In fact, the two are so intrinsically linked that organizations desiring to become \u201cAI-driven\u201d now need a single architected platform that tightly integrates two broad classes of functionality:<\/p>\n
\n- Data management functions \u2013<\/strong>\u00a0from collecting enterprise data to managing it in storage, via modern data governance, pipelining, orchestration, lineage, quality, and privacy<\/li>\n
- Analytic functions \u2013<\/strong>\u00a0including data exploration, model design, machine learning, and agile CI\/CD cycles to produce evolving LLMs<\/li>\n<\/ul>\n
Databricks as a Data Intelligence Platform<\/h4>\n
For years, Databricks has been bringing data management and advanced analytic processing together, by integrating several open-source and proprietary technologies, including Apache Spark, Delta Lake, MLflow, Python, Presto, Trino, DBRX, and more, available via serverless cloud services. These have recently coalesced into Databricks\u2019 concept of the\u00a0Data Intelligence Platform<\/strong>, and you should expect DAIS to hit this concept repeatedly, along with the many components required of the concept.<\/p>\nThis newly conceived Data Intelligence Platform is essentially a melding of:<\/p>\n
\n- Databricks Lakehouse \u2013<\/strong>\u00a0a cornucopia of data management functionality<\/li>\n
- DatabricksIQ \u2013<\/strong>\u00a0based on Databricks\u2019 acquisition of the analytic tool MosaicML<\/li>\n<\/ul>\n
The architecture of the resulting Data Intelligence Platform is unified by (among other things):<\/p>\n
\n- Single governance layer \u2013<\/strong>\u00a0all data and AI functionality, across full data lifecycle<\/li>\n
- Single query engine \u2013<\/strong>\u00a0satisfying ETL, SQL, ML, and BI requirements<\/li>\n<\/ul>\n
![](\"https:\/\/media1-production-mightynetworks.imgix.net\/asset\/1716c62c-35da-4718-8da7-ed2029b20e70\/1717078979451.png?ixlib=rails-4.2.0&fm=jpg&q=75&auto=format&w=1400&h=1400&fit=max&impolicy=ResizeCrop&constraint=downsize&aspect=fit\")
<\/a>A Reference Architecture for the Data Intelligence Platform<\/strong>
\nSource:\u00a0https:\/\/www.databricks.com\/product\/data-intelligence-platform<\/a><\/p>\nManaging Data at Scale in an AI World<\/h4>\n
We have all seen the recent ascension of artificial intelligence (AI) \u2013 whether generative, predictive, or otherwise \u2013 such that AI is now much more usable, functional, and critical to business success than with prior generations. Hence, you should expect DAIS to hit AI hard from a data intelligence platform perspective, along with related techniques and products for machine learning (ML), large language models (LLMs), and retrieval-augmented generation (RAG).<\/p>\n
Anyone leaving DAIS (or any vendor-sponsored data conference) should take home the vendor\u2019s current and future strategies in two key areas where AI and data management are increasingly melding:<\/p>\n
The data intelligence platform\u2019s ability to deliver AI-ready data, at scale<\/strong>\u00a0\u2013 In other words, a data management platform must continue to deliver data that\u2019s appropriate to business requirements and technical standards on a per use case basis. Yet, data platforms are now under pressure to deliver data and analytic processing at the scale, performance, source diversity, and multi-model tolerance that use cases in AI demand.<\/p>\nThe AI-driven functionality embedded within tools for data management and analytics\u00a0<\/strong>\u2013 The point of extending D&A tools this way is to provide greater automation and intelligence, which can lift the productivity of data engineers and data scientists, plus give production systems smarter monitoring, automatic data remediation, and quicker reactions. Given the shrinking time span allotted for the design, development, and deployment of analytic solutions, AI-driven automation is becoming a survival strategy for CI\/CD cycles in analytics.<\/p>\nA Cornucopia of Compelling Conference Content<\/h3>\n
DAIS is jam-packed, with over 500 sessions \u2013 yikes!. This includes a giant, diverse collection of keynotes, presentations, user case studies, product demos, speaker panels, roundtables, vertical industry sessions, training courses, certification, social events, and networking opportunities.<\/p>\n
For example, here\u2019s a sampling of topics that DAIS sessions will cover. (Warning: This is a very long list!)<\/p>\n
\n- Delta Lake Architectures and Best Practices<\/li>\n
- Data Quality in Data Lakehouses<\/li>\n
- Apache Parquet for Lakehouses<\/li>\n
- Data Mesh for Data Lakes<\/li>\n
- Cost Optimization for Data Lakehouses<\/li>\n
- What is Lighthouse?<\/li>\n
- Data Engineering Methods for D&A<\/li>\n
- Data prep for AI, GenAI, and other advanced forms of analytics<\/li>\n
- MLOps and LLMOps<\/li>\n
- Attaining AI\/ML Maturity<\/li>\n
- AutoML and MLflow,<\/li>\n
- Women in Data<\/li>\n
- Vector Data and Search<\/li>\n
- Managing and Analyzing Hybrid Data<\/li>\n
- LLMs and Segment Anything Models<\/li>\n
- LLMs and Hallucination Risk<\/li>\n
- Retrieval-Augmented Generation (RAG)<\/li>\n
- Unity Catalog and Data Governance<\/li>\n
- Geospatial Data on Databricks<\/li>\n
- Data and Applications Migrations to Databricks<\/li>\n
- Business Value and ROI for AI Applications<\/li>\n
- Data Pipelines and Streams on Databricks<\/li>\n
- Data Warehousing with Databricks SQL<\/li>\n<\/ul>\n
One of the compelling traits of DAIS is that it includes literally dozens of case study presentations given by actual Databricks-using organizations. This differs from many other \u201cuser conferences\u201d where (ironically) there aren\u2019t many user speakers. Here is a sampling of user organizations participating in case study sessions: (Again, the list is long!)<\/p>\n
\n- General Motors, Lexmark, Sight Machine, GrabTaxi, JetBlue, Idaho National Laboratory, Corning, Unilever, CVS Health, Kroger, Northern Trust, Western Governors University, Fox, Kwik Trip, Ordnance Survey, VisitBritain, AccuWeather, Workday, Trek Bicycle, Royal Caribbean Group, Instacart, J P Morgan Chase, Northwestern Mutual, Doordash, Shell, Blackberry, Albertsons, AT&T, US Department of State, Bayer AG, Rolls-Royce, Chevron Phillips, Bloomberg, Rivian, Michelin, Starbucks, T-Mobile, Nike, Comcast, Bridgestone, Boeing, Capital One, McDonald\u2019s, DraftKings, etc.<\/li>\n<\/ul>\n
Another admirable trait of DAIS is the inclusion of several speaker panels and other sessions devoted to vertical industries, including: Public Sector; Energy; Financial Services; Manufacturing and Transportation; Retail and Consumer Goods; etc.<\/p>\n
Finally, DAIS is organized into eight or so tracks, to help attendees find sessions and topics that suit their needs and interests. Tracks include:<\/p>\n
\n- Data Lakehouse Architecture<\/li>\n
- Data Engineering and Streaming<\/li>\n
- Data Governance<\/li>\n
- Generative AI<\/li>\n
- Data Science and Machine Learning<\/li>\n
- Data Strategy and Lakehouse Implementation<\/li>\n
- Data Warehousing, Analytics, and BI<\/li>\n<\/ul>\n
Conclusion and Recommendations<\/h3>\n
Again, there 500+ sessions at Databricks\u2019 Data + AI Summit 2024 (DAIS), and they are distributed over four days. This means that no one can attend them all. Hence, you should get organized and create a list of \u201cfavorites\u201d before arriving at DAIS. That way, you won\u2019t miss sessions relevant to you.<\/p>\n
You can peruse the gigantic agenda for DAIS 2024 online at:\u00a0https:\/\/www.databricks.com\/dataaisummit\/agenda<\/a><\/p>\n
Databricks as a Data Intelligence Platform<\/h4>\n
For years, Databricks has been bringing data management and advanced analytic processing together, by integrating several open-source and proprietary technologies, including Apache Spark, Delta Lake, MLflow, Python, Presto, Trino, DBRX, and more, available via serverless cloud services. These have recently coalesced into Databricks\u2019 concept of the\u00a0Data Intelligence Platform<\/strong>, and you should expect DAIS to hit this concept repeatedly, along with the many components required of the concept.<\/p>\n This newly conceived Data Intelligence Platform is essentially a melding of:<\/p>\n The architecture of the resulting Data Intelligence Platform is unified by (among other things):<\/p>\n We have all seen the recent ascension of artificial intelligence (AI) \u2013 whether generative, predictive, or otherwise \u2013 such that AI is now much more usable, functional, and critical to business success than with prior generations. Hence, you should expect DAIS to hit AI hard from a data intelligence platform perspective, along with related techniques and products for machine learning (ML), large language models (LLMs), and retrieval-augmented generation (RAG).<\/p>\n Anyone leaving DAIS (or any vendor-sponsored data conference) should take home the vendor\u2019s current and future strategies in two key areas where AI and data management are increasingly melding:<\/p>\n The data intelligence platform\u2019s ability to deliver AI-ready data, at scale<\/strong>\u00a0\u2013 In other words, a data management platform must continue to deliver data that\u2019s appropriate to business requirements and technical standards on a per use case basis. Yet, data platforms are now under pressure to deliver data and analytic processing at the scale, performance, source diversity, and multi-model tolerance that use cases in AI demand.<\/p>\n The AI-driven functionality embedded within tools for data management and analytics\u00a0<\/strong>\u2013 The point of extending D&A tools this way is to provide greater automation and intelligence, which can lift the productivity of data engineers and data scientists, plus give production systems smarter monitoring, automatic data remediation, and quicker reactions. Given the shrinking time span allotted for the design, development, and deployment of analytic solutions, AI-driven automation is becoming a survival strategy for CI\/CD cycles in analytics.<\/p>\n DAIS is jam-packed, with over 500 sessions \u2013 yikes!. This includes a giant, diverse collection of keynotes, presentations, user case studies, product demos, speaker panels, roundtables, vertical industry sessions, training courses, certification, social events, and networking opportunities.<\/p>\n For example, here\u2019s a sampling of topics that DAIS sessions will cover. (Warning: This is a very long list!)<\/p>\n One of the compelling traits of DAIS is that it includes literally dozens of case study presentations given by actual Databricks-using organizations. This differs from many other \u201cuser conferences\u201d where (ironically) there aren\u2019t many user speakers. Here is a sampling of user organizations participating in case study sessions: (Again, the list is long!)<\/p>\n Another admirable trait of DAIS is the inclusion of several speaker panels and other sessions devoted to vertical industries, including: Public Sector; Energy; Financial Services; Manufacturing and Transportation; Retail and Consumer Goods; etc.<\/p>\n Finally, DAIS is organized into eight or so tracks, to help attendees find sessions and topics that suit their needs and interests. Tracks include:<\/p>\n Again, there 500+ sessions at Databricks\u2019 Data + AI Summit 2024 (DAIS), and they are distributed over four days. This means that no one can attend them all. Hence, you should get organized and create a list of \u201cfavorites\u201d before arriving at DAIS. That way, you won\u2019t miss sessions relevant to you.<\/p>\n You can peruse the gigantic agenda for DAIS 2024 online at:\u00a0https:\/\/www.databricks.com\/dataaisummit\/agenda<\/a><\/p>\n\n
\n
![](\"https:\/\/media1-production-mightynetworks.imgix.net\/asset\/1716c62c-35da-4718-8da7-ed2029b20e70\/1717078979451.png?ixlib=rails-4.2.0&fm=jpg&q=75&auto=format\")
<\/a>A Reference Architecture for the Data Intelligence Platform<\/strong>
\nSource:\u00a0https:\/\/www.databricks.com\/product\/data-intelligence-platform<\/a><\/p>\nManaging Data at Scale in an AI World<\/h4>\n
A Cornucopia of Compelling Conference Content<\/h3>\n
\n
\n
\n
Conclusion and Recommendations<\/h3>\n