Organizational Principles for Cloud Data Architecture Teams

Two broad classes of people have a vested interest in cloud data architectures:  Technical data specialists (who design and deploy the data architecture, plus populate it with data) and business end users (who consume the data provisioned by the data architecture). The managers of these user constituencies are also stakeholders, since they tend to control budgets and human resources that are needed to support CDAs. 

Technical Data Specialists: 

• Data Architects: Design a cloud data architecture as a series of connected data design patterns. They also establish data standards (for quality, modeling, integration preferences). 
• Data Engineers: Do most of the actual building, deployment, and optimizing of the cloud data architecture, based on the architects’ designs, specifications, and other guidance. Note that data engineers handle a wide range of data management skills, including data integration, pipelining, data quality, mastering data, and metadata and other data semantics. 

Business End Users Who Consume Data: 

• Managers who control budgets: Want to see how their money will be spent on building new data architectures, plus maintaining them once in production.  
• Business End Users Who Consume Data: Need to express their data-driven business requirements, so the architecture team will provide them with the data and tools they need to do their jobs well. 

Note that all these people have varying amounts of influence on platform choices, such as the selection of data clouds, DBMSs, various data management tools, end-user tools for reporting and analytics, and so on. Since all these people are stakeholders, in some sense, they or their representatives should be involved in CDA planning and execution. 

Principles for Cloud Data Architecture Project Planning 

Here are tips for the most important elements of a project plan for designing, building, and migrating data solutions in the context of a CDA:

• Start by evangelizing the CDA concept, plus lining up sponsors, budgets, and buy-in from multiple teams. 
• Design a CDA that addresses the requirements of high priority use cases. Expect the design to evolve, as the project moves forward. 
• Focus technical efforts on data integration, because it is the path to the cloud data architecture (CDA). 
• Also focus on the end-user requirements for data consumption (reporting and analytics), since the quality of the new CDA will be judged by how well you satisfy these. 
• Assure success for important execution steps: Requirements gathering, architecture design, migration to cloud, building new architectural elements, reengineering solutions after the lift and shift. 
• Put “lift and shift” in its place. It is useful for making a quick beachhead on cloud, but should be followed by re-engineering and other improvements to data and solutions, once these are on cloud.  
• Hire consultants to execute the migration of data and analytics solutions to cloud. Consultants may also design the new CDA; select tools and platforms for it; and provide support and maintenance over time. 
• Assemble a diverse team for the migration to cloud of data and analytics. It should include all stakeholders, not just technical staff. 
• Create a multi-phase project plan, not a risky big bang project. Phase One should be a low-risk, high-value business use case. 
• Prioritize the components of your CDA to create a list of project phases. 
• Do not just move data and solutions to cloud. Improve them, too. 
• Revise and improve your data architecture, once on cloud, so it is optimized for cloud technology and it satisfies the most recent version of business requirements. 

Recommendations 

• Stay focused on the data: Don’t be distracted by the many other components within a cloud data architecture. After all, the whole point is to collect and deliver complete, quality data in the right condition, at the right moment. 
• Keep your CDA’s software portfolio lean: Otherwise, its licenses can ravage your budget and burn up lots of resources for training, staffing, and development. 
• Stress the interoperability of the CDA’s many tools and platforms: Otherwise, it becomes a low-value bucket of silos. 
• Take data governance very seriously: Compliance infractions with your CDA’s data can be devastating for your organization. 
• Take data governance very seriously: Without it, the data stores of your CDA can deteriorate into useless data swamps. 
• Create data standards for your CDA: These include data quality metrics, preferred modeling paradigms, coding rules, naming conventions, and metadata and cataloging rules. 
• Embrace cloud’s differences: Most of these are beneficial, such as less platform maintenance, easy to attain speed and scale, simpler data modeling, and consumption paced pricing. 
• Don’t just move to cloud. Improve to cloud: Find the time and resources to improve dataset quality and completeness, metadata and cataloging semantics, and solution relevance per current business requirements. 
• Do more that “lift and shift” when you migrate data to cloud: It’s good to get on the new cloud as early as possible, but expect to double back and improve data once it’s there. 
• Start with valued business use cases, when you plan your CDA: Be sure these are supported well by your CDA, since these are what business people want and therefore how they will judge your CDA. 
• Know the common CDA design patterns and how they relate to your requirements: If data science is a priority use case, you probably need a CDA built around a lake. If business reporting and dashboards are the priority, then a data warehouse focused paradigm may be more appropriate. 
• Be careful with consumption-based pricing: Early on, get in the habit of monitoring and documenting cloud resource usage, then correlating that to your bills from the cloud provider. Even if you have workloads from on premises, they will burn resources differently on cloud. 
• Create a multi-phase project plan for the new CDA: Avoid a risky big bang project. The first phase should be a low-risk but valuable use case, so you can demonstrate value on cloud, as early as possible.