Overview
This input type allows you to specify log files in an S3 bucket for Edge Delta to monitor. With this action, S3 will be monitored via SQS.
Note
To use this input type, you must have SQS notifications enabled on the target bucket.
To learn more, see Enable SQS Notification.
Enable SQS Notification
To use this input type, you must have SQS notifications enabled on the target bucket.
- Create an SQS queue to receive messages from S3. The SQS queue must have a policy that allows S3 to send messages to the SQS queue. Review the following example of a policy:
{
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "__owner_statement",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<account id>:root"
},
"Action": "SQS:*",
"Resource": "arn:aws:sqs:us-west-2:<account id>:"
},
{
"Sid": "s3_send_statement",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": [
"SQS:SendMessage"
],
"Resource": "arn:aws:sqs:us-west-2:<account id>:my-sqs",
"Condition": {
"ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:my-bucket" },
"StringEquals": { "aws:SourceAccount": "<account id>" }
}
}
]
}
-
Configure the S3 bucket to send notifications to the SQS queue.
-
To learn how to configure the S3 bucket, review this document from Amazon.
-
-
Create a new IAM user with programmatic access type. The Edge Delta agent will use this user to access SQS and S3. Review the following example of a policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"sqs:DeleteMessage",
"s3:GetObject",
"sqs:DeleteMessageBatch",
"sqs:ReceiveMessage"
],
"Resource": [
"arn:aws:s3:::my-bucket/*",
"arn:aws:sqs:us-west-2:<account id>:my-sqs"
]
}
]
}
-
Create an access key for the IAM user.
-
You will need the access key ID and secret in the next step.
-
-
Add S3 input to the agent.
Review Parameters
Review the following parameters that you can configure in the Edge Delta App:
| Visual Editor | YAML | Description |
| S3 Input Labels | labels |
Enter a descriptive label for this input. When you create a workflow, you will use this label to enter your input into the workflow. errorcheck is the default label. This parameter is required. Review the following example: - labels: "errorcheck" |
| SQS URL | sqs_url |
Enter the full URL for the SQS queue. This parameter is required. Review the following example: sqs_url: "https://sqs.us-west-2.amazonaws.com/233765244907/taylan-test-sqs" |
| AWS Access Key ID | access_key_id |
Enter the access key ID that the agent will use to pull SQS and download S3 files. This access key ID must have the following permissions:
This parameter is required. Review the following example: access_key_id: "ABCDEFG" |
| AWS Access Secret | access_secret |
Enter the access secret key that corresponds with the access key ID. This parameter is required. Review the following example: access_secret: "Dn2djaskl" |
| Role ARN | role_arn |
Enter the ARN that has permissions to use the desired IAM Role To learn more, review this document from AWS. This parameter is optional. Review the following example: role_arn: "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>" |
| External ID | external_id |
Enter the external ID associated with the desired IAM role. To learn more, review this document from AWS. This parameter is optional. Review the following example: external_id: "053cf606-8e80-47bf-b849-8cd1cc826cfc" |
| AWS Region | region |
Enter the region where the bucket and SQS queue are located. This parameter is required. Review the following example: region: "us-west-2" |
| Log Type | log_type |
Select (or enter) the desired log types to monitor:
This parameter is optional. Review the following example: log_type: alb |
| Filters | filters |
Select an existing filter to add to this input. To learn how to create a filter, see Filters. This parameter is optional. Review the following example: filters: |
Review Sample Configuration
Review the following sample configuration:
s3_sqs:
- labels: "errorcheck"
sqs_url: "https://sqs.us-west-2.amazonaws.com/233765244907/taylan-test-sqs"
access_key_id: "ABCDEFG"
access_secret: "Dn2djaskl"
region: "us-west-2"
- labels: "alb"
sqs_url: "https://sqs.us-west-2.amazonaws.com/233765244907/my-alb-logs-sqs"
access_key_id: "ABCDEFG"
access_secret: "Dn2djaskl"
region: "us-west-2"
log_type: alb
- labels: "sqs-inout-assumes-role"
sqs_url: "https://sqs.us-west-2.amazonaws.com/233765244907/taylan-test-sqs"
region: "us-west-2"
role_arn: "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>"
external_id: "053cf606-8e80-47bf-b849-8cd1cc826cfc"
Quick links