Innovating ELT Data Pipelines with Inflight Data Formatting on Hevo

The ELT process has modernized data pipelines and has become the preferred technique over traditional ETL. It has fastened your data loading speed and facilitated efficient data analysis. 

However, there is still a delay in the analysis and the time to obtain reports and insights, as your analysts have to run a few additional jobs at the warehouse to clean and format the data before running queries on it. 

This is because the data loaded into the warehouse may not be consistent, organized, and per the data warehouse tables format. 

If you eliminate these data formatting jobs at the data warehouse, you will further increase the efficiency of your data pipeline. All you need is a method to format and clean the data before loading it to the warehouse.  

Thus, we innovated and launched an Inflight Data Formatter on Hevo Data Pipelines, where data teams can perform lightweight data formatting inflight.

What is Inflight Data Formatting?

Inflight Data Formatting is an ELT data pipeline feature innovated by Hevo, i.e., implementing lightweight data formatting within the pipeline before loading the data to the warehouse.

Hevo’s Inflight Data Formatter cleans, enriches, and standardizes data on the fly without impacting the load performance of the pipeline.

We believe inflight data formatting should be essential in your modern ELT data pipelines. It enables your data teams to effortlessly maintain a consistent data format at the warehouse.

Why should you perform inflight data formatting on Hevo?

Setting up inflight data formatting on Hevo powers up your modern data analytics stack and offers a number of benefits to your analytics process. Following are the key benefits,

Data Consistency at Warehouse

Data from different sources can be formatted in a single format for your data warehouse. Your data teams can easily manage data-type mismatches and ensure the data is always compatible with the data warehouse tables. 

Faster Analytics

It speeds up your analysis by eliminating the need to run additional transformations at the warehouse. Analysts get real-time access to consistent and organized data.

Setting up an ELT data pipeline with inflight data formatting is the fastest way to move data for analytics.

Legacy Data Management

Usually, data from your legacy data sources will not be compatible with your modern data warehouse. But with Inflight Data Formatter, your team can easily load legacy data to a warehouse by formatting the legacy data inflight as per the data warehouse structure. 

Key functions you can set up on Hevo’s Inflight Data Formatter

Your data team can perform several lightweight data formatting or transformation functions on Hevo. They can enrich, split, merge, normalize data, and handle data-type mismatch.

Hevo's Inflight Data Formatter - Key Functions

Following are the most commonly used inflight formatting jobs that your team can set up on Hevo, 

1. Data and Time Formatting 

Multiple data sources record dates and times in different formats. However, it’s efficient to maintain a single format in your data warehouse. You can format your date and time from source to any format inflight. 

2. Mask or Hash Data

Your data may include a few sensitive fields like your customers’ personal information. This should not be accessible by your business users. You can mask or hash the sensitive fields like email address, contact number, etc., on the fly. 

3. Data Enrichment

You can enrich your data by adding new fields inflight. For example, adding the product name for the respective dataset from the source while loading it to the warehouse. 

4. JSON Normalization

You can convert or flatten JSON objects from NoSQL databases into a relational database. Functions like parsing JSON packages, formatting JSON objects into rows and columns, loading nested data into tables, and setting up relationships between those tables can be performed on the fly using inflight data formatting.

5. Clean and Filter Events

There are a few events or fields that you would not want to load to your warehouse. You can filter out such fields or events in the pipeline itself. For example, filtering out all the canceled orders from your sales dataset. 

Here is an example of how Inflight Data Formatter boost ELT data pipelines

Consider a global e-commerce company with multiple versions of their website like .us, .au for each location. They want to build a dashboard to analyze accumulated and region-wise traffic on each product page. 

They would need to load the data for each region from Google Analytics to their data warehouse while adding the respective location name for each record. This can be easily achieved by creating a pipeline with inflight data formatting for each property on Google Analytics. 

Hevo's Inflight Data Formatter-Example

Using Inflight data formatting on Hevo, they can add a field ‘Country’ and populate it with the respective country name for each region. For a pipeline loading data from .au, they can set the rule to add a new field ‘Country’ with populated value ‘Australia’. 

With this, analysts will have access to complete data of website traffic for each region with their respective location names in the data warehouse. Thus, they can directly run queries to build a dashboard for product pages while adding country as a dimension. 

Powerful and Flexible Inflight Data Formatting Console

Hevo provides a flexible and powerful console to set up inflight data formatting. There are 2 ways to set up inflight data formatting on Hevo, one is by using our drag-and-drop interface, and the other is by using the Python console.

Hevo's Inflight Data Formatter-Dashboard

Using a drag-and-drop interface, you can effortlessly set up required formatting in minutes without writing any code. 

You can set up and write your own rule or logic for data formatting using the Python console

Here is what our customers say about our Inflight Data Formatter

Hevo's Inflight Data Formatter - Customer Testimonial 1

Try out Hevo’s Inflight Data Formatter

Check out how inflight data formatting further optimizes your data pipelines and data stack by signing up for a 14-day free trial

If you are a Hevo customer and you want to set up inflight data formatting then you can simply go to your respective pipeline’s transformation section and set up your respective rule or function.


Source link

Tags: No tags

Add a Comment

Your email address will not be published. Required fields are marked *