Overview
This input type allows you to specify log files in an S3 bucket for Edge Delta to monitor. With this action, S3 will be monitored via SQS.
Note
To use this input type, you must have SQS notifications enabled on the target bucket.
To learn more, see Enable SQS Notification.
Enable SQS Notification
To use this input type, you must have SQS notifications enabled on the target bucket.
- Create an SQS queue to receive messages from S3. The SQS queue must have a policy that allows S3 to send messages to the SQS queue. Review the following example of a policy:
{ "Version": "2008-10-17", "Id": "__default_policy_ID", "Statement": [ { "Sid": "__owner_statement", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<account id>:root" }, "Action": "SQS:*", "Resource": "arn:aws:sqs:us-west-2:<account id>:" }, { "Sid": "s3_send_statement", "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": [ "SQS:SendMessage" ], "Resource": "arn:aws:sqs:us-west-2:<account id>:my-sqs", "Condition": { "ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:my-bucket" }, "StringEquals": { "aws:SourceAccount": "<account id>" } } } ] }
-
Configure the S3 bucket to send notifications to the SQS queue.
-
To learn how to configure the S3 bucket, review this document from Amazon.
-
-
Create a new IAM user with programmatic access type. The Edge Delta agent will use this user to access SQS and S3. Review the following example of a policy:
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "sqs:DeleteMessage", "s3:GetObject", "sqs:DeleteMessageBatch", "sqs:ReceiveMessage" ], "Resource": [ "arn:aws:s3:::my-bucket/*", "arn:aws:sqs:us-west-2:<account id>:my-sqs" ] } ] }
-
Create an access key for the IAM user.
-
You will need the access key ID and secret in the next step.
-
-
Add S3 input to the agent.
Review Sample Configuration
Review the following sample configuration:
s3_sqs: - labels: "errorcheck" sqs_url: "https://sqs.us-west-2.amazonaws.com/233765244907/taylan-test-sqs" access_key_id: "ABCDEFG" access_secret: "Dn2djaskl" region: "us-west-2" - labels: "alb" sqs_url: "https://sqs.us-west-2.amazonaws.com/233765244907/my-alb-logs-sqs" access_key_id: "ABCDEFG" access_secret: "Dn2djaskl" region: "us-west-2" log_type: alb - labels: "sqs-inout-assumes-role" sqs_url: "https://sqs.us-west-2.amazonaws.com/233765244907/taylan-test-sqs" region: "us-west-2" role_arn: "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>" external_id: "053cf606-8e80-47bf-b849-8cd1cc826cfc"
Review Parameters
Review the following parameters that you can configure in the Edge Delta App.
labels
Required
Enter a descriptive label for this input.
When you create a workflow, you will use this label to enter your input into the workflow.
errorcheck is the default label.
Review the following example:
- labels: "errorcheck"
sqs_url
Required
Enter the full URL for the SQS queue.
Review the following example:
sqs_url: "https://sqs.us-west-2.amazonaws.com/233765244907/taylan-test-sqs"
access_key_id
Required
Enter the access key ID that the agent will use to pull SQS and download S3 files.
This access key ID must have the following permissions:
- sqs:DeleteMessage
- sqs:DeleteMessageBatch
- sqsReceiveMessage
- s3:GetObject
Review the following example:
access_key_id: "ABCDEFG"
access_secret
Required
Enter the access secret key that corresponds with the access key ID.
Review the following example:
access_secret: "Dn2djaskl"
role_arn
Optional
Enter the ARN that has permissions to use the desired IAM Role
To learn more, review this document from AWS.
Review the following example:
role_arn: "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>"
external_id
Optional
Enter the external ID associated with the desired IAM role.
To learn more, review this document from AWS.
Review the following example:
external_id: "053cf606-8e80-47bf-b849-8cd1cc826cfc"
region
Required
Enter the region where the bucket and SQS queue are located.
Review the following example:
region: "us-west-2"
log_type
Optional
Enter the desired log types to monitor:
- alb
- cloudtrail
Review the following example:
log_type: alb
filters
Optional
Enter an existing filter to add to this input.
To learn how to create a filter, see Filters.
Review the following example:
filters:
- info