PB-scale data real-time analysis solution: Architecture practice based on Tencent Cloud native data lake warehouse

Introduction (pain point analysis)

As a data engineer or architect, are you struggling with the following questions?

Data silos and governance difficulties.Business data is scattered in dozens of data sources such as MySQL, Kafka, log files, CSV, etc., with different formats, which makes it difficult to unify management and guarantee data quality.
Costs are out of control.To cope with cyclical business peaks (e.g., big promotions, events), traditional big data platforms (e.g., self-built Hadoop) need to configure hardware resources according to the peak demand, resulting in idle resources most of the time and high costs.
Performance bottlenecks.The traditional ETL process is complex and lengthy, and it often takes T+1 days from data entry to report output, which can't support the business' urgent need for real-time data insights, and decision-making is always one step slower.
Complex architecture and heavy operations and maintenance.Maintaining the stable operation of a complete set of big data clusters (HDFS, Hive, Spark, Presto) requires a large investment in professional operation and maintenance manpower, high technical threshold, and difficult troubleshooting.

If you are struggling with the above problems, then this article will provide you with a complete solution based on Tencent Cloud's native data lake warehouse to realize efficient, economical, and unified real-time analysis of petabytes of data.

Solution Architecture Diagram and Overview

architecture diagram

PB-scale data real-time analysis solution: Architecture practice based on Tencent Cloud native data lake warehouse - LikaCloud

Overview.

The core of this program is“Segregation of accounts”cap (a poem)“Harmonized metadata management”. All raw data are uniformly deposited in a highly reliable, low-costTencent Cloud Object Storage (COS), forming the cornerstone of the data lake.Data Lake Computing (DLC)As the brain, it is responsible for unified metadata management, permission control and data governance, eliminating the need to build your own Hive Metastore.Elastic MapReduce (EMR)As a powerful compute engine, it pulls up clusters on-demand and analyzes data in COS directly through standard Spark, Presto, and other compute frameworks, releasing resources when the task is complete. Ultimately, the analysis results can be directly consumed by BI tools, data applications or AI platforms.

Value Proposition.This solution perfectly solves the pain points in the introduction, reduces storage and computation costs through storage-computation separation, breaks down data silos through unified metadata, realizes fast analysis through elastic Serverless computation engine, and minimizes O&M complexity.

Core Products and Components

Component name. Tencent Cloud Object Storage (COS)
- Playing the role.integrated architecturePersistent Storage TierThe system stores all raw data, processed data and calculation results.
- Key configuration/selection recommendations.Thermal data that require frequent analysis are analyzed usingStandard StorageThe use of cold data for archivingArchive Storage, leveraging lifecycle strategies for automatic transitions to maximize cost savings.
- Why choose it.Provides unlimited capacity and 99.99999999999% data reliability, making it the ideal base for building data lakes. Seamlessly integrates natively with EMR and DLC for excellent performance optimization.
Component name. Data Lake Computing (DLC)
- Playing the role.structured“Intelligent Brain”It provides unified metadata management, data rights and access control, SQL data catalog and Serverless interactive query services.
- Key configuration/selection recommendations.Use Serverless mode directly, no need for pre-built resources. Easily interface with data on COS and define table structures through its data catalog function.
- Why choose it.It completely solves the problem of metadata silos. EMR, BI tools, etc. can access the unified metadata view through DLC to realize the consistent management of permissions and table structure. Its Serverless Spark capability can realize senseless submission of Spark jobs, greatly simplifying operation and maintenance.
Component name. Elastic MapReduce (EMR)
- Playing the role.coreElastic Compute Engine, responsible for running large-scale data processing tasks (e.g. ETL, interactive queries, machine learning).
- Key configuration/selection recommendations.optionvolumetric billingandElastic ScalabilityMode, automatically scaling up and down Task nodes based on CPU/memory load. Options for co-located deployment with COS, DLC for optimal performance.
- Why choose it.It provides full-stack capabilities of the open source big data ecosystem (Hadoop, Spark, Presto, Hbase, etc.) and is deeply integrated with Tencent Cloud out of the box. Its elasticity capability ensures efficient utilization of resources, paying only for the actual amount of computation used.

Summary of program benefits

? Extreme cost optimization.Storage-computing separation, on-demand scaling of computing resources, compared to traditional self-built fixed clusters.Comprehensive cost can be reduced by more than 50%。
⚡ Efficient Analytics and Agile Ops.No need for data migration, EMR can directly analyze COS data at high speed; DLC unified metadata management thatMinute-by-minute building of big data platforms, O&M workload dropped by 90%.
? Breaking down data silos.A copy of the data is stored in the COS and shared and analyzed by multiple computational engines (EMR, cloud functions, etc.) through the unified view of the DLC, truly enabling theData Inclusion。
? ️ Enterprise level security governance.The DLC provides column-level data rights control and seamless integration with CAM.Audit logComplete to meet enterprise data security and compliance requirements.

Application Scenarios and Applicable Customers

Typical application scenarios.
- Interactive Instant Query.Data analysts use Presto/Spark SQL to perform fast queries on massive historical data directly through DLC and get instant results.
- Real-time log analysis.Business logs are written to COS/Kafka in real time and analyzed by EMR Streaming or Spark Streaming for near real-time processing to monitor business status.
- Machine Learning and Data Mining.Use EMR's Spark on k8s cluster to read training data directly from COS for large-scale model training.
Applicable customer characteristics.
- The volume of data has reachedTB to PB leveland sustained growth of traditional and Internet companies.
- receive in no small measureExpensive scaling and performance bottlenecks in traditional data warehousesTroubled team.
- wishIntegration of multiple data sourcesCustomers who are building a unified view of their enterprise's data.
- Team HopeFocus on business data development rather than underlying infrastructure operations and maintenance。