Elastic Search Solutions: Cloud-Native Search Engine Elasticsearch Service Supports Peak Business Access at Low Cost

Introduction (pain point analysis)​

For backend developers and architects of fast-growing content platforms, e-commerce sites or enterprise applications, a powerful and flexible search engine is the core infrastructure for improving user experience and business efficiency. However, when building your own or using traditional search services on the cloud, are you often plagued by these problems:

• Traffic peaks are difficult to cope with.When encountering big promotions and hot events, the query QPS instantly spikes, and the fixed cluster resources are overwhelmed, resulting in slow response or even downtime of the search service, which directly affects user conversion and business revenue.
• It is difficult to balance resources and costs.To ensure stability during peak periods, you have to allocate resources to the highest traffic, but during the vast majority of flat periods, a large number of compute and storage resources sit idle, resulting in significant wasted costs.
• High O&M complexity.Cluster expansion and contraction, version upgrade, index management, performance tuning and other operation and maintenance work requires the investment of a professional team and a lot of energy, the technical threshold is high, which distracts the focus on the core business.

​If you are struggling with the stability, high cost and heavy O&M of search services, then this article will provide you with a complete cloud-native elastic search solution.​

​Solution Architecture Diagram and Overview​

​Objective.A picture is worth a thousand words and gives the reader a big picture view.

​Architecture Diagram.

Architecture Overview.​

The user's application (Web/App/server) initiates a search request, which is first distributed via Tencent Cloud Load Balancing (CLB), and then the request accesses the Tencent Cloud Elasticsearch (ES) service via a secure private network (VPC).The ES cluster uses theServerless ArchitectureOrElastic scaling groups based on monitoring metricsThe ES cluster can be scaled up in seconds according to the real-time query pressure on the compute nodes. Data is written in real time through components such as Kafka, and inside the ES cluster through theIntelligent hot and cold separationThe mechanism stores hot data with high-frequency access on SSD disks to guarantee performance, automatically sinks cold data with low-frequency access into object storage (COS), and further reduces storage costs through ZSTD compression technology. The entire solution achieves the ultimate cost optimization while guaranteeing millisecond response and high availability.

​Value Proposition.Adoption of the ProgramExtreme elasticityCope with peak traffic byIntelligent Layering and CompressionReducing integrated costs throughFully Managed ServicesEliminating the burden of O&M perfectly solves all the pain points raised in the introduction.

​Core Products and Components​

• Component name.​ ​Tencent Cloud Elasticsearch Service (ES)​
  • Playing the role.As the core of the entire solution, a distributed search and analytics engine service is provided to handle all data indexing and query requests.
  • Key configuration/selection recommendations.​
    • Architecture Selection.Highly recommended for scenarios where traffic fluctuates wildly.Serverless version, it truly realizes pay-as-you-go and millisecond elastic scaling. For scenarios with regular fluctuations, you can choose the elastic scaling version and set up a scaling policy based on CPU utilization or query QPS.
    • Node Configuration.Recommended to enableexclusive master nodeto improve cluster stability; data nodes useHigh IOOrgeneral purposeinstances with SSD cloud drives for optimal I/O performance.
    • Function Configuration.Be sure to turn onIndexing Lifecycle Management (ILM)function to automate hot-temperature-cold stratification; enable theZSTD compressionfor higher compression ratios and faster compression speeds.
  • Why choose it.Tencent Cloud ES is a 100%-compatible open source Elasticsearch fully managed service that provides all the capabilities of open source, as well as value-added features such as enterprise-grade security, monitoring and alerts, and one-click upgrades, so that users don't need to care about the underlying infrastructure, and only need to focus on the search business logic itself.
• ​Component name.​ ​Tencent Cloud Object Storage (COS)​
  • Playing the role.Serves as the underlying storage for the ES Cold Data Tier, which holds low-frequency accessed archived data and provides highly reliable, very low-cost storage services.
  • Key configuration/selection recommendations.Creating for ES Cold DataStandard StorageOrLow Frequency StorageBuckets are sufficient, with a durability of up to 99.999999999999% (12 9s), with no additional configuration required.
  • ​Why choose it.COS has a much lower storage cost than high-performance block storage, and by seamlessly integrating with ES's ILM functionality, cold data can be senselessly sunk into this, reducing storage costs by more than 80% without losing data traceability.
• Component name.​ ​Tencent Cloud Load Balancing (CLB)​
  • Playing the role.As a unified entrance to the service, it automatically distributes massive search requests to multiple ES data nodes in the backend, eliminating single points of failure and providing capabilities such as HTTPS offloading.
  • Key configuration/selection recommendations.optionApplied CLB, configured as a TCP/UDP listener, forwarded to port 9200 of the ES node. It is recommended to bind the elastic public IP and configure the bandwidth limit.
  • Why choose it.Achieve load balancing of access traffic, improve service availability and scalability, while shielding the internal details of the back-end cluster to the client, easier to manage.

​Summary of program benefits​

• ⚡ Extreme elasticity with second-by-second scaling.Adopting Serverless or elastic scaling group, it can be instantly expanded to the required scale according to the business pressure, and the resources are automatically recovered after the traffic peak, truly realizing pay-per-use.
• 💸 Significantly reduce overall costs.Through intelligent hot and cold data separation and ZSTD compression technology, it reduces the cost of cold data storage by more than 80% and avoids the waste generated by reserving a large amount of resources to cope with the peak, and the comprehensive cost can be optimized to 50%+.
• 🛡️ Out-of-the-box, enterprise-grade high availability.It provides 99.95% service availability SLA, built-in replica mechanism, cross-availability deployment, automatic failover capability, and security features such as VPC network isolation, security groups, access logs, etc. to ensure business continuity and data security.
• 🔧 Operationally free and focused on business innovation.Fully managed service eliminates the tedious operation and maintenance work such as hardware deployment, software installation, version upgrade, fault recovery, etc. You only need to make simple configuration through the console or API, you can get powerful search capabilities, so that the team can focus more on the core business development.

​Application Scenarios and Applicable Customers​

• Typical application scenarios.​
  • E-commerce website/app.Product search, filtering, recommended lists, to cope with 618, double eleven and other big promotions during the instantaneous massive query.
  • Content and information platforms.Keyword search and full-text search for news, videos, articles and more.
  • Log Analysis and Operations Monitoring (ELK).Collect, store and analyze massive log data from enterprise servers and applications in real time for troubleshooting and business insights.
  • Intra-enterprise data retrieval.Such as CRM, ERP and other systems within the customer, order, product information query.
• Applicable customer characteristics.​
  • There are significant peaks and valleys in the business load and the peaks are unpredictable.
  • The amount of data is huge (TB-PB level) and the data access has a clear hot and cold characteristic.
  • Medium to large enterprises that are looking for cost-effective solutions and want to optimize their infrastructure costs.
  • Lack of a professional Elasticsearch Ops team for all developers who want an out-of-the-box, stable and reliable search service.

​Related links​