Financial-grade data governance solutions: building an enterprise-level data blood and quality control system

Introduction (pain point analysis)

Dear CTOs, data directors and architects of financial firms, in the wave of digital transformation, are you deeply troubled by the following questions?

The Data Bloodline Black Hole.When regulatory requirements trace the source of a report's data, it takes days to manually sort through it, and it's impossible to quickly and clearly answer the question "where did the data come from and what was done with it".
The quality of the data is worrisome.Key business indicators have inconsistent data in different systems, and dirty data and duplicate data seriously affect the accuracy of risk control decisions and business reports, but there is a lack of effective monitoring and auditing tools.
Sensitive data is out of control.Sensitive information such as customer ID numbers and bank card numbers are scattered all over the place, and who is accessing them and how they flow is completely opaque, facing huge compliance risks and leakage hazards.
Compliance pressures are enormous.Meeting regulatory requirements such as the Data Security Law and the Bank of China's Financial Data Security Classification Guidelines requires investing a large amount of manpower in data classification and grading and security audits, which is costly and inefficient.

If you are looking for a solution to build a data governance system that meets financial regulatory requirements, Aliyun's financial-grade data governance solution based on DataWorks and Data Security Center (DSC) will provide you with a clear and efficient path.

Solution Architecture Diagram and Overview

The following figure shows the overall architecture and data flow relationship of data governance built based on AliCloud DataWorks and DSC:

Financial-grade data governance solutions: building an enterprise-level data blood and quality control system - LikaCloud

The core design ideas of this program are"Governance as process, security as built-in"The workflow is as follows:

1.Data Access and Development.pass (a bill or inspection etc)DataWorksThe Data Integration module synchronizes scattered data sources (RDS, MaxCompute, OSS, etc.) to the data warehouse and completes the configuration and scheduling of data processing tasks through a visual development interface.
2.Automatic blood discovery.DataWorks automatically parses the data processing tasks (SQL, ETL, etc.) to generate a report from the data source to the final report.End-to-end data lineage mappingand presented in the data map.
3.Sensitive data identification and classification hierarchy. Data Security Center (DSC)With built-in financial industry templates and machine learning algorithms, sensitive data (e.g., customer PII information, transaction information, etc.) is automatically scanned, discovered, and categorized and tagged with classification hierarchies.
4.Quality auditing and monitoring.Configure data quality monitoring rules in DataWorks to verify the timeliness, completeness, and accuracy of key data tables, and automatically alert and block downstream tasks when problems occur to guarantee the quality of data output.
5.Harmonize asset management and security controls.All metadata, bloodlines, quality scores, and security labels are aggregated into a unified enterprise data asset catalog. Unified data security policies (e.g., desensitization, access control) are formulated and enforced based on risk identification results and audit logs from DSC.

The value proposition of the architecture is.It transforms the core links of data governance (metadata, data quality, data security) from the traditional manual, passive and isolated mode to an automated, active and collaborative system that opens up the whole chain, so that the data can really become a credible, controllable and usable strategic asset.

Core Products and Components

component name	play a role	Key configuration/selection recommendations	Why choose it
Big Data Development Governance Platform DataWorks	Core platform for data development and governance. Provides one-stop functions such as data integration, data development, data quality, data mapping, data services, etc., and is the main host platform for data governance processes.	-Version Selection.Financial companies are advised to choose the Enterprise Edition to meet higher performance and security requirements. -Data quality.Configure integrity, uniqueness, and accuracy rules and set blocking thresholds for core business metrics and tables. -Data Map.Turn on automatic bloodline resolution and regularly maintain asset descriptions and Owner information.	It seamlessly integrates data production and data governance processes, avoiding the problem of "two skins" between governance and development. Its automatic bloodline discovery capability far exceeds that of open source tools, greatly reducing operation and maintenance costs.
Data Security Center (DSC)	Data Security Governance Core Engine.. Responsible for automated sensitive data discovery, classification and grading, risk detection, auditing and desensitization, it is the technical cornerstone of data security governance.	-Scanning range.Configure periodic scanning tasks for core data warehouses (e.g. MaxCompute), databases (RDS). -Identify the template.Choose and customize financial industry templates to accurately identify ID cards, bank cards, phone numbers and other sensitive information. -Risk modeling.Turn on risk detection models for abnormal access, high-frequency operations, etc.	It utilizes machine learning technology to solve the problem of "not being able to find" sensitive information in massive data. It is deeply integrated with DataWorks, MaxCompute and other products to realize the unified landing of security policies.
Cloud-Native Big Data Computing Services MaxCompute	Big Data computing engine.. As an enterprise-level data warehouse, storing and computing full-volume data, it is the main object and vehicle for data governance.	-Resource planning.Purchase prepaid CUs or use volume-based billing based on data volume and growth expectations. -Data Lifecycle.Develop a reasonable strategy for tiered data storage and lifecycle management to optimize costs.	Providing EB-class storage and high-performance computing power, fully managed, stable and reliable, it is the cornerstone of building an enterprise digital warehouse. Seamless integration with DataWorks is inherent.
Ant Privacy Computing Service Platform	Privacy Enhanced Computing Platform(Optional). Provides secure and reliable data fusion computing capabilities in scenarios where data collaboration is required but the original data is not expected to be out of domain.	-Usage Scenarios.It is suitable for business scenarios that require data collaboration with external organizations, such as joint marketing and joint risk control.	Under the premise of meeting data security and compliance requirements, unlocking the value of data and realizing "data availability and invisibility" is an advanced form of data governance.

Summary of program benefits

? Full-link data bloodline.Automated parsing generates field-level data pedigree mapping, supporting impact analysis and root cause tracing, making the data lineage clear at a glance.
✅ Proactive data quality audits.Provide strong rules monitoring and intelligent early warning, changing after-the-fact remediation to before-the-fact prevention and guaranteeing accurate and reliable data.
? ️ Intelligent sensitive data protection.ML-based automatic discovery and classification and grading of sensitive data, combined with precise access control and desensitization policies, to meet financial compliance requirements.
? Harmonize asset value insights.Build a unified enterprise data asset catalog, aggregating metadata, quality scores, security labels, hotness and other information, data assets at a glance.
⚙️ Automation of governance processes.Embed governance specifications (e.g., quality rules, security policies) into the development process to realize "governance left", improve efficiency and reduce human costs.

Application Scenarios and Applicable Customers

This solution is ideal for the following business scenarios and customers:

Application Scenarios.
- Meet regulatory compliance.Meet the audit requirements of financial industry regulators for data security, data quality, and data lineage.
- Improvement of data quality.Solve the problems of inconsistent, inaccurate and untimely data that have long plagued the business, and improve decision-making efficiency.
- Data security building.Build enterprise data security protection system to prevent sensitive data leakage and protect customer privacy.
- Data Assetization.Sort out and revitalize enterprise data assets to improve the efficiency of finding, understanding and using data.
Applicable Customers.
- Financial institutions such as banks, securities and insurance.Strongly regulated industry with a rigid need for data governance.
- Large business groups.The complexity of the business, the multiplicity of systems and the confusion of data urgently require unified governance.
- Data-sensitive Internet companies.For example, e-commerce and social platforms, which have a large amount of user data, need to be strictly controlled.