Subscribe to the Teradata Blog

Get the latest industry news, technology trends, and data science insights each week.



I consent that Teradata Corporation, as provider of this website, may occasionally send me Teradata Marketing Communications emails with information regarding products, data analytics, and event and webinar invitations. I understand that I may unsubscribe at any time by following the unsubscribe link at the bottom of any email I receive.

Your privacy is important. Your personal information will be collected, stored, and processed in accordance with the Teradata Global Privacy Policy.

Data Mesh and the City Planner

Data Mesh and the City Planner
City planning is a complex beast. Planners must predict population growth and its impact on zoning, utilities, transportation, social services, and regulations to name a few. A city planner rarely, if ever, gets a chance to start from scratch. They must take the current architecture and infrastructure into consideration when planning the future. 

Data mesh planning is a lot like city planning. Planners want to provide as much freedom and flexibility as possible to encourage business growth. To do so, they must provide zoning regulations, common infrastructure services and governance capability, which promote both efficient and sustainable growth. 

Zoning Regulations
City Planning Data Mesh Planning
  • Land zone types; public, commercial, residential, manufacturing, etc.
  • Zone rules; signage, parking, landscaping, mailboxes
  • Domain boundaries e.g., POS Sales, Online Sales, Sales Aggregations, Product Recommendations
  • Data product rules e.g., data quality, interoperability, security, metadata
 
 
Cities provide borders and rules for land management via zoning regulations. Likewise, data mesh provides borders for data products via business domain boundaries and rules for data products via data management practices. Each domain team is responsible for providing reliable, high-quality information to everyone within the data mesh community. They must instill confidence of their data products by ensuring good data management practices such as data quality, security, discoverability, accessibility, usability, and interoperability, are consistently applied.

Infrastructure Services
City Planning Data Mesh Planning
  • Utilities e.g., water, electricity, gas, broadband
  • Services e.g., sanitation, police, fire, medical, schools
  • Accessibility e.g., roads, bridges, mass transit
  • Infrastructure e.g., cloud XaaS / marketplace and on-premises capabilities
  • Services e.g., metadata, infosec, observability
  • Accessibility e.g., APIs (and abstracted views for co-located and connected data products)
 
 
Cities provide common services in the form of public utilities, roads, mass transportation, and emergency services, e.g., local fire, police and medical. Shared infrastructure and social services facilitate more efficient development and maintenance of businesses and homes. This in turn enables citizens and businesses to be more productive. 

Data mesh provides infrastructure services which support data product development and operations via a self-serve data platform. Each organization can subscribe to common services from the cloud marketplace or build their own. An example of a common need is metadata management. Data product teams are responsible for the creation and maintenance of all their technical, operational, and business metadata. Collaborative metadata, which captures information like consumer ratings and feedback, is created and maintained by the data product consumers. Planners must provide clarity on how metadata will be managed, internally within and externally from data products, and how those services will be provided. Other potential candidates for common services might be around information security, observability, and reference data management.   

Governance
City Planning Data Mesh Planning
  • Federal, state and city compliance e.g., water safety, state and federal highways, inspectors
  • Financial governance i.e., budgeting, funding, procurement management
  • Regulatory compliance e.g., PII, PCI, PHI
  • Data Governance e.g., policies and enforcement methods
  • Financial governance i.e., budgeting, funding, procurement management
 
 
The city planner must also consider governance polices and compliance. The city planner must be aware of federal, state and city policies and regulations, clarify the rationale behind new or modified city regulation requests, and understand their enforcement capabilities. If there are holes in the regulations or gaps within the enforcement capabilities, regulations will be ineffective, and issues will emerge. For example, unsafe drinking water, zone misuse, transportation gridlock and city fund mismanagement erode public trust and can result in costly litigation.

Data mesh, likewise, requires governance to ensure the trust and confidence that the data products are being delivered via federated governance. This means regulations, standards and best practices must be clearly defined and published to ensure everyone is keenly aware of overall governing principles and their roles and responsibilities; and secondly, but just as important, processes to verify their compliance. For example, the ability to detect PII, PCI, PHI and other regulator violations is critical to organizations financially, socially, and ethically.

Organizations, just as cities, also have financial constraints. In the subscription services model, financial governance is critical to continuously track resource spend, identify potential offenders, and proactively address them before they cause an overrun of the budget. For example, automated monitoring and management of resources via workload management capabilities can dynamically prioritize the needs of the user community without negatively impacting their productivity. Observability capabilities can identify biggest offenders, which drive opportunities to refactor code or redesign data structures to be more performant. Without these types of capabilities, resources can dynamically scale up and out of control.

Like city planners, data mesh planners are not starting from scratch. Within the analytic ecosystem, most of these concepts have existed for years. Some organizations are leveraging these mature data management concepts effectively and may not choose to change. For those who do decide to transition to data mesh, there is no data mesh easy button. And mergers and acquisitions may also find themselves straddling a traditional and data mesh approach. In either case, both approaches will co-exist within organizations for some time. 

Transitioning to data mesh is a journey based on business-driven initiatives and delivering business value. Planners must provide clear guidelines on zone regulations, infrastructure services and governance to ensure efficient and sustainable growth within the overall community.  
Portrait of Dwayne Johnson

(Author):
Dwayne Johnson

Dwayne Johnson is a principal ecosystem architect at Teradata, with over 20 years' experience in designing and implementing enterprise architecture for large analytic ecosystems. He has worked with many Fortune 500 companies in the management of data architecture, master data, metadata, data quality, security and privacy, and data integration. He takes a pragmatic, business-led and architecture-driven approach to solving the business needs of an organization.
  View all posts by Dwayne Johnson

Turn your complex data and analytics into answers with Teradata Vantage.

Contact us