Replication your data from a variety of sources using our Generic Webhooks handler

Webhooks

Many platforms support sharing of data using Webhooks. Often, Webhooks provide real-time data as changes happen in the source system. Historical data is typically not available using Webhooks, but some sources may offer this capability. We suggest getting a manual data dump from the source to load historical data for sources configured via webhooks.

Challenges with Webhooks 

Detecting Data Types 

Sources may not always publish or honor the published documentation when it comes to data types. Therefore it is important to accurately detect or infer the data type based on the incoming data stream and create tables in the warehouse with the columns that have the most accurate data type. 

Schema Control 

A Webhook is essentially a way for an application to provide data or information to another application. It is a one-way data sharing that is triggered by events in the system. Typically, webhook endpoints receive data in JSON, CSV, or XML; JSON being the most common method. When a source is sending data to Webhooks, it is possible that the data structure in which the data is shared changes over time. Columns may be added, columns may be removed, nested columns may be added, or removed, and many other such scenarios. It is thus important for Daton to gracefully handle these changes to the schema without the user having to worry about making changes to their pipeline. 

Handling Nested Objects

Nested data when added to a webhook data stream must be handled efficiently. Depending on the warehouse, the nested data may have to be flattened, or loaded as a JSON in the warehouse. 

Loading Methods

Daton supports Append-Only mode of loading for Webhook data. Although primary keys may be available in the source, the data in the warehouse will only be appended to the tables configured using the webhooks. User must de-duplicate the data prior to using the data for reporting or analytics needs.

Steps to configure a Webhook

Step 1 - Select an Integration Name

This integration name will be used to identify the Webhook in the Daton integrations screens. 

Step 2 - Enter a Table Name

Enter a table name. Daton will use this input to create a table in the warehouse. Please make sure that the table name is unique and not already in use in the data warehouse. 

Step 3 - Generate a Webhook URL

Daton auto-generates a URL that can be used to configure the Webhooks. Use the Copy to clipboard functionality to copy the URL and use it in your code or within the source application.

If you have any questions about setting up Webhooks, please reach out to us on Chat or email us at support@sarasanalytics.com