Rehydrations

George Alpizar
George Alpizar
  • Updated

Overview

You can use this document to learn about the Rehydrations page in the Edge Delta App. 


Concepts

Rehydration is the process of pushing already-archived data to a streaming destination, such as Splunk, Elasticsearch, etc.

Note

Azure Blob and LocalStorage are not supported for rehydration. 


Related Concepts

If you have sensitive data that cannot leave your infrastructure or cannot be viewed by the Edge Delta backend, then you can create an on-prem rehydration.


Instructions


Create a Rehydration

You can use these instructions to learn how to create a rehydration hosted on Edge Delta's platform.

Note

To create a rehydration, you must have an archived output and streaming output already configured in your account.

Note

If you have sensitive data that cannot leave your infrastructure or cannot be viewed by the Edge Delta backend, then you can create an on-prem rehydration.

To learn more, see Set Up On-Prem Rehydration.

  1. In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Rehydrations
  2. Click Create.
  3. In Tag, select the tag for the agent configuration that handles the logs that you want to rehydrate. 
    • Based on the tag you select, the Source Type field will populate with options. 
  4. In From and To, select a date and time range of when data was collected to send to the streaming destination.
    • This configuration represents the time frame for when data is collected, not when data is archived; however, the time difference between collection and archiving is a few minutes. 
  5. In Archive Source, select an existing archive output whose data you want to rehydrate and send to a streaming output.

    Note For S3 Users

    Before you created an S3 archive output, you first needed to have created an IAM user, and then attached a custom policy. That policy contained the 3 permissions, PutObject, GetObject, and ListBucket. 

    If you created an S3 archive for rehydration purposes only, then at a minimum, your custom policy only needs to contain the GetObject permission. 

    All other permissions are only required for archiving purposes. 

    As a result, if you prefer, you can create 2 different S3 archive integrations with different custom policies. 

  6. In Destination, select an existing streaming output to send the archived data. 
  7. (Optional) In Keyword, enter a term to filter specific logs that will be sent.
    • Only logs that match the entered keyword will be pushed to the streaming destination.
    • This field is compatible with a valid Go regex.
    • If you leave this field empty, then all logs will be pushed to the streaming destination.
  8. In Host, select a host. This section is populated based on the agent configuration tag that you selected. 
  9. In Source Type, select a source type. This section is populated based on the agent configuration tag that you selected. 
  10. If your organization has a configured rehydration limit, then you must click Analyze. You will not be able to click Create until you click Analyze
    • This action will display how much data will be pushed to the streaming destination. If your individual rehydration configuration is above the organization’s rehydration limit, then you will not be able to create the rehydration.
      • To troubleshoot, you can update the settings for the individual rehydration. Or, if you have the correct account permissions, you can update your organization’s rehydration settings. To do so, return to the Rehydrations page, click Settings, and then update the data limits. 
    • If you click Analyze and you receive the following message, then please contact Edge Delta Support to troubleshoot: Failed to analyze rehydration data size. Failed to get rehydration analysis. Please have available your organization name and the time when the message appeared. 
  11. Click Create
    • The app may take a few minutes to display your newly created rehydration.
    • After you create a rehydration, you can click on the entry to view details. 
      • Note: The Invalid Lines entry means that there may be empty files. 

Configure Limits for Rehydrations

You can use this instructions to configure universal limits for your rehydrations. Specifically, you can make configurations to prevent duplicated rehydration requests of the same data, which will prevent duplicated log entries. 

Note

A blank field indicates that the setting does not have a limit. 

  1. In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Rehydrations
  2. Click Settings.
  3. For Is On-Prem, mark true if your rehydrations are on-prem rehydrations. 
    • If your rehydrations are hosted on Edge Delta's platform, then mark false.
  4. For Maximum Rehydration Size, enter a limit on the total size of the files to rehydrate, and then select the size type (byte, kilobyte, megabyte, gigabyte).
  5. For Maximum Concurrent Rehydration Count, enter the maximum number of in-progress / non-completed rehydrations that can run at any time, per organization. 
  6. For Maximum Concurrent Rehydration Count Per User, enter the maximum number of in-progress / non-completed rehydrations that can run at any time, per user. 
  7. Click Save

Share this document