Connecting Elasticsearch to S3: 4 Easy Steps

Are you trying to derive deeper insights from your Elasticsearch by moving the data into a larger Database like Amazon S3? Well, you have landed on the right article. Now, it has become easier to replicate data from Elasticsearch to S3.

This article will give you a brief overview of Elasticsearch and Amazon S3. You will also get to know how you can set up your Elasticsearch to S3 integration using 4 easy steps. Moreover, the limitations of the method will also be discussed in further sections. Read along to know more about connecting Elasticsearch to S3 in the further sections.

Table of Contents

Introduction to Elasticsearch

ElasticSearch Logo
Image Source

Elasticsearch accomplishes its super-fast search capabilities through the use of a Lucene-based distributed reverse index. When a document is loaded to Elasticsearch, it creates a reverse index of all the fields in that document. A reverse index is an index where each of the entries is mapped to a list of documents that contains them. Data is stored in JSON form and can be queried using the proprietary query language.

Elasticsearch has four main APIs – Index API, Get API, Search API and Put Mapping API. Index API is used to add documents to the index. Get API allows to retrieve the documents and Search API enables querying over the index data. Put Mapping API is used to add additional fields to an already existing index. 

The common practice is to use Elasticsearch as part of the standard ELK stack, which involves three components – Elasticsearch, Logstash, and Kibana. Logstash provides data loading and transformation capabilities. Kibana provides visualization capabilities. Together, three of these components form a powerful Data Stack. 

Behind the scenes, Elasticsearch uses a cluster of servers to deliver high query performance. An index in Elasticsearch is a collection of documents. Each index is divided into shards that are distributed across different servers. By default, it creates 5 shards per index with each shard having a replica for boosting search performance. Index requests are handled only by the primary shards and search requests are handled by both the shards. 

The number of shards is a parameter that is constant at the index level. Users with deep knowledge of their data can override the default shard number and allocate more shards per index. A point to note is that a low amount of data distributed across a large number of shards will degrade the performance. 

Scaling in Elasticsearch is accomplished by adding more servers. The architecture can automatically rebalance the data and query load across available nodes. Fault tolerance in Elasticsearch is accomplished through cross-cluster replication. A remote cluster can be set up to sync with the primary cluster and serve as hot standby.

Amazon offers a completely managed Elasticsearch service that is priced according to the number of instance hours of operational nodes. 

To know more about Elasticsearch, visit this link.

Introduction to Amazon S3

Amazon S3 Logo
Image Source

AWS S3 is a fully managed object storage service that is used for a variety of use cases like hosting data, backup and archiving, data warehousing, etc. Amazon handles all operational activities related to capacity scaling, pre-provisioning, etc and the customers only need to pay for the amount of space that they use. It offers comprehensive access controls to meet any kind of organizational and business compliance requirements through an easy-to-use control panel interface. 

S3 supports analytics through the use of AWS Athena and AWS redshift spectrum through which users can execute SQL queries over data stored in S3. S3 buckets can be encrypted by S3 default encryption. Once enabled, all items in a particular bucket will be encrypted. 

S3 achieves high availability by storing the data across a number of distributed servers. Naturally, there is an associated propagation delay with this approach and S3 only guarantees eventual consistency. But, the writes are atomic; which means at any time, the API will return either the new data or old data and never will it provide a corrupted response. 

Conceptually S3 is organized as buckets and objects. A bucket is the highest level S3 namespace and acts as a container for storing objects. They have a critical role in access control and usage reporting is always aggregated at the bucket level. An object is the fundamental storage entity and consists of the actual object as well as the metadata. An object is uniquely identified by a unique key and a version identifier. 

Customers can choose the AWS regions in which their buckets need to be located according to their cost and latency requirements. A point to note here is that objects do not support locking and if two PUTs come at the same time, the request with the latest timestamp will win. This means if there is concurrent access, users will have to implement some kind of locking mechanism on their own. 

To know more about Amazon S3, visit this link.

Hevo Data, an Automated No-code Data Pipeline, helps you directly transfer data from Elasticsearch and S3 to Business Intelligence tools, Data Warehouses, or a destination of your choice in a completely hassle-free & automated manner. Hevo’s end-to-end Data Management connects you to Elasticsearch’s cluster using the Elasticsearch Transport Client and synchronizes the data available in the cluster to your preferred data warehouse using indices. Hevo’s Pipeline allows you to leverage the services of both Generic Elasticsearch & AWS Elasticsearch.

Moreover, Hevo enables you to load data from files in an S3 bucket and make it a part of your Destination database or Data Warehouse seamlessly. Hevo’s Data pipeline automatically unzips any Gzipped files on ingestion and files are re-ingested in case there is any data update.

Hevo is fully managed and completely automates the process of not only loading data from 100+ data sources (including 40+ free sources) but also enriching the data and transforming it into an analysis-ready form without having to write a single line of code. Its fault-tolerant architecture ensures that the data is handled in a secure, consistent manner with zero data loss. It provides a consistent & reliable solution to manage data in real-time and always have analysis-ready data in your desired destination.

Check out what makes Hevo amazing:

  • Secure: Hevo has a fault-tolerant architecture that ensures that the data is handled in a secure, consistent manner with zero data loss.
  • Auto Schema Mapping: Hevo takes away the tedious task of schema management & automatically detects the schema of incoming data from S3 buckets and Elasticsearch files and maps it to the destination schema.
  • Quick Setup: Hevo with its automated features, can be set up in minimal time. Moreover, with its simple and interactive UI, it is extremely easy for new customers to work on and perform operations.
  • Transformations: Hevo provides preload transformations through Python code. It also allows you to run transformation code for each event in the Data Pipelines you set up. You need to edit the event object’s properties received in the transform method as a parameter to carry out the transformation. Hevo also offers drag and drop transformations like Date and Control Functions, JSON, and Event Manipulation to name a few. These can be configured and tested before putting them to use for aggregation.
  • Hevo Is Built To Scale: As the number of sources and the volume of your data grows, Hevo scales horizontally, handling millions of records per minute with very little latency.
  • Incremental Data Load: Hevo allows the transfer of data that has been modified in real-time. This ensures efficient utilization of bandwidth on both ends.
  • Live Support: The Hevo team is available round the clock to extend exceptional support to its customers through chat, email, and support calls.

With continuous Real-Time data movement, load your data from Elasticsearch & S3 to your destination warehouse with Hevo’s easy-to-setup and No-code interface. Try our 14-day full access free trial.

Get Started with Hevo for Free

Steps to Connect Elasticsearch to S3 using Custom Code

Moving data from Elasticsearch to S3 can be done in multiple ways. The most straightforward is to write a script to query all the data from an index and write it into a CSV or JSON file. But the limitations to the amount of data that can be queried at once make that approach a nonstarter. You will end up with errors ranging from time outs to too large a window of query. So, you need to consider other approaches to connect Elasticsearch to S3.

Logstash, which is a core part of the ELK stack, is a full-fledged data load and transformation utility. With some adjustment of configuration parameters, it can be made to export all the data in an elastic index to CSV or JSON. The latest release of log stash also includes an S3 plugin, which means the data can be exported to S3 directly without intermediate storage. Thus, Logstash can be used to connect Elasticsearch to S3. Let us look in detail into this approach and its limitations.

Using Logstash

Image Source

Logstash is a service side pipeline that can ingest data from a number of sources, process or transform them and deliver to a number of destinations. In this use case, Log stash input will be Elasticsearch and output will be a CSV file. Thus, using logstash, Elasticsearch to S3 integration can be easily done.

Logstash is based on data access and delivery plugins and is an ideal tool for connecting Elasticsearch to S3. For this exercise, you need to install the Logstash Elasticsearch plugin and the Logstash S3 plugin. Below is a step-by-step procedure to connect Elasticsearch to S3:

  1. Execute the below command to install logstash Elasticsearch plugin.
    logstash-plugin install logstash-input-elasticsearch
  2. Execute the below command to install logstash output s3 plugin.
    logstash-plugin install logstash-output-s3
  3. Next involves creating a configuration for the logstash execution. An example configuration to execute this is provided below.


    input {  elasticsearch {     hosts => "elastic_search_host"     index => "source_index_name"     query => '     {     "query": {     "match_all": {}     }     }    '   } } output {    s3{      access_key_id => "aws_access_key"      secret_access_key => "aws_secret_key"      bucket => "bucket_name"    } }

    In the above JSON, replace the elastic_search_host with the URL of your source Elasticsearch instance. The index key should have the index name as the value. The query tries to match every document present in the index. Remember to also replace the AWS access details and the bucket name with your required details.

  4. Create this configuration and name it as es_to_s3.conf

  5. Execute the configuration using the following command.
    logstash -f es_to_s3.conf

    The above command will generate JSON output matching the query in the provided S3 location. Depending on your data volume, this will take a few minutes. There are multiple parameters that can be adjusted in the S3 configuration to control variables like output file size etc. A detailed description of all config parameters can be found here.

    By following the above-mentioned steps, you can easily connect Elasticsearch to S3.

Limitations of Connecting Elasticsearch to S3 using Custom Code

The above approach is the simplest way to transfer data from an Elasticsearch to S3 without using any external tools. But it does have some limitations. Below are to limitations that are associated while setting up Elasticsearch to S3 integrations:

  1. This approach to connect Elasticsearch to S3 works fine for a one-time load, but in most situations, the transfer is a continuous process that needs to be executed based on an interval or triggers. To accommodate such requirements, customized code will be required.
  2. This approach to connect Elasticsearch to S3 is resource-intensive and can hog the cluster depending upon the number of indexes and the volume of data that needs to be copied. 


This article provided you with a comprehensive guide to Elasticsearch and Amazon S3. You got to know about the methodology to connect Elasticsearch to S3 using Logstash and its limitations as well. Now, you are in the position to connect Elasticsearch to S3 on your own.

The manual approach of connecting Elasticsearch to S3 using Logstash will add complex overheads in terms of time, and resources. Such a solution will require skilled engineers and regular data updates. Furthermore, you will have to build an in-house solution from scratch if you wish to transfer your data from Elasticsearch or S3 to a Data Warehouse for analysis.

Hevo Data provides an Automated No-code Data Pipeline that empowers you to overcome the above-mentioned limitations. Hevo caters to 100+ data sources (including 40+ free sources) and can seamlessly transfer your S3 and Elasticsearch data to the Data Warehouse of your choice in real-time. Hevo’s Data Pipeline enriches your data and manages the transfer process in a fully automated and secure manner without having to write any code.

Learn more about Hevo

Want to take Hevo for a spin? Sign up for a 14-day free trial and experience the feature-rich Hevo suite firsthand.

What are your thoughts on moving data from Elasticsearch to S3? Have you explored other approaches that have worked for you? Let us know in the comments.

Source link

Tags: No tags

Add a Comment

Your email address will not be published. Required fields are marked *