Low Cost Big Data Storage and Compute Solution: Object Storage OSS + Compute Separation Architecture Reduces Costs 50%

Introduction (pain point analysis)

Are you facing any of the following challenges as a big data platform owner?

Storage costs have skyrocketed.The data volume is growing at a rate of 50% per year, and traditional HDFS storage solutions require constant server expansion, making hardware procurement and server room operation and maintenance costs a heavy burden.
Waste of computing resources.In order to cope with occasional computing peaks (e.g. month-end report generation, annual audit), a huge Hadoop/Spark cluster must be maintained for a long period of time, resulting in an average CPU utilization of less than 20% and a serious waste of resources.
Scalability bottlenecks.Storage and computation are tightly coupled, and when expanding storage, you must expand the computation node at the same time, which is complicated and cannot realize the independent elastic scaling of resources.
Complexity of technical operations and maintenance.Self-built clusters require a specialized team to perform continuous version upgrades, troubleshooting and performance tuning, which has a high technical threshold and distracts the energy that should be devoted to data business innovation.

One sentence summary.If you are struggling with rising big data infrastructure costs and heavy operations and maintenance work, then this article will provide you with a complete solution based on AliCloud's storage-computing separation architecture to realize cost reduction and efficiency.

Solution Architecture Diagram and Overview

architecture diagram

Low Cost Big Data Storage and Compute Solution: Object Storage OSS + Compute Separation Architecture Reduces Costs 50% - LikaCloud

Overview.

The core of this program is"Segregation of accounts"together with"Serverlessization". All data is deposited directly into theAliCloud Object Storage (OSS), utilizing its unlimited capacity and low-cost tiered storage capabilities (standard, low-frequency, archival) as a persistent storage pedestal for data lakes. Computational tasks are performed byElastic Container Instance (ECI)cap (a poem)E-MapReduce Serverlessetc. are hosted by serverless engines, which are only pulled up in seconds while the task is running, paid for by the amount of computing resources actually used (CPU/memory/runtime duration), and released as soon as the task is completed. The whole process is event-driven (e.g. new file upload to OSS) and there is no need to manage any servers.

Value Proposition.This solution hits the pain point, by transforming the high fixed cluster cost into very low storage cost + on-demand computing cost, the comprehensive cost can be reduced by more than 50% and completely liberate the pressure of operation and maintenance.

Core Products and Components

Component name. AliCloud Object Storage (OSS)
- Playing the role.integrated architectureCore Storage CornerstoneThe data is carried in the same way as all the other data.
- Key configuration/selection recommendations.
  - Frequently accessed hot data.adoptionStandard StorageType.
  - Temperature data for occasional visits.adoptionlow-frequency accessStorage type (low access cost, even lower storage cost).
  - Cold data for archiving/backup.adoptionplace on fileOrcold archivingStorage type (lowest cost).
  - By configuring theLife cycle rulesThe system enables automatic conversion of data from standard -> low-frequency -> archive to maximize cost savings.
- Why choose it.Provides 12 9 data persistence at 1/3 or less the cost of self-built hard disk storage, ideal for store-computer separation architectures.
Component name. Elastic Container Instance (ECI)
- Playing the role. On-demand elastic computing core. Used to run custom containerized computing tasks (e.g., Python scripts, custom data handlers).
- Key configuration/selection recommendations.
  - with regards toShort-term, suddenof computational tasks (e.g., running ETL for 1-2 hours per day), prioritize the use of ECI.
  - Configured according to the vCPU and memory specifications required for the task, it supports small-size instances of 0.25 cores to avoid wasting resources.
  - Viaevent trigger(e.g., OSS file upload events) automatically wake up computing resources to realize a fully automated pipeline.
- Why choose it.It truly realizes "per-second billing and on-demand scaling" of computing resources, without the need to reserve resources, which greatly improves the utilization rate of resources.

Summary of program benefits

? The combined cost is a straight 50%+.With low-cost OSS for storage and pay-as-you-go for computing, there is no need to pay for idle resources, and the total cost of ownership (TCO) drops dramatically compared to self-built fixed clusters.
⚡ Extreme elasticity with second-by-second scaling.In the face of data floods or sudden analytical demands, computing resources can be expanded instantly without the need for advance procurement and deployment, greatly enhancing business agility.
? ️ Highly available and maintenance free.AliCloud Infrastructure Services provide high availability SLAs, eliminating the need to care about the failure and maintenance of the underlying infrastructure, and allowing the team to focus on the data development itself.
? Openness and compatibility.Fully compatible with the open source ecosystem, existing data processing programs can be smoothly migrated to protect the existing technology investment.

Application Scenarios and Applicable Customers

Typical application scenarios.
- Cyclical ETL assignments.Data cleansing, transformation and loading operations performed on a daily/weekly basis.
- Interactive Instant Query.Data analysts initiate occasional query tasks where computational resources are created with the start of the query and released with the end of the query.
- Event-driven processing.For example, as soon as a new log file is uploaded to OSS, the anomaly detection or report generation task is immediately triggered.
Applicable customer characteristics.
- possessCost-sensitiveof companies and teams.
- Calculated demand existsClear peaks and troughs(e.g., scenarios with many daytime tasks and few nighttime tasks).
- wishBuilding a Big Data Platform from Scratchand companies that don't want to build a large O&M team.
- in progressDigital Transformation, traditional businesses looking to introduce big data capabilities with minimal trial and error costs.