S3 via SQS (AWS) Inputs

George Alpizar
George Alpizar
  • Updated

Overview

This input type allows you to specify log files in an S3 bucket for Edge Delta to monitor. With this action, S3 will be monitored via SQS.

Note

To use this input type, you must have SQS notifications enabled on the target bucket. 

To learn more, see Enable SQS Notification.

Enable SQS Notification

To use this input type, you must have SQS notifications enabled on the target bucket. 

  1. Create an SQS queue to receive messages from S3. The SQS queue must have a policy that allows S3 to send messages to the SQS queue. Review the following example of a policy: 
{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<account id>:root"
      },
      "Action": "SQS:*",
      "Resource": "arn:aws:sqs:us-west-2:<account id>:"
    },
    {
	   "Sid": "s3_send_statement",
	   "Effect": "Allow",
	   "Principal": {
	    "Service": "s3.amazonaws.com"  
	   },
	   "Action": [
	    "SQS:SendMessage"
	   ],
	   "Resource": "arn:aws:sqs:us-west-2:<account id>:my-sqs",
	   "Condition": {
	      "ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:my-bucket" },
	      "StringEquals": { "aws:SourceAccount": "<account id>" }
      }
    }
  ]
}

  1. Configure the S3 bucket to send notifications to the SQS queue.

  2. Create a new IAM user with programmatic access type. The Edge Delta agent will use this user to access SQS and S3. Review the following example of a policy:  

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "sqs:DeleteMessage",
                "s3:GetObject",
                "sqs:DeleteMessageBatch",
                "sqs:ReceiveMessage"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket/*",
                "arn:aws:sqs:us-west-2:<account id>:my-sqs"
            ]
        }
    ]
}

  1. Create an access key for the IAM user.

    • You will need the access key ID and secret in the next step.

  2. Add S3 input to the agent.

Review Sample Configuration

Review the following sample configuration: 

  s3_sqs:
    - labels: "errorcheck"
      sqs_url: "https://sqs.us-west-2.amazonaws.com/233765244907/taylan-test-sqs"
      access_key_id: "ABCDEFG"
      access_secret: "Dn2djaskl"
      region: "us-west-2"

    - labels: "alb"
      sqs_url: "https://sqs.us-west-2.amazonaws.com/233765244907/my-alb-logs-sqs"
      access_key_id: "ABCDEFG"
      access_secret: "Dn2djaskl"
      region: "us-west-2"
      log_type: alb

    - labels: "sqs-inout-assumes-role"
      sqs_url: "https://sqs.us-west-2.amazonaws.com/233765244907/taylan-test-sqs"
      region: "us-west-2"
      role_arn: "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>"
      external_id: "053cf606-8e80-47bf-b849-8cd1cc826cfc"

Review Parameters

Review the following parameters that you can configure in the Edge Delta App. 

labels

Required

Enter a descriptive label for this input. 

When you create a workflow, you will use this label to enter your input into the workflow. 

errorcheck is the default label.

Review the following example:

- labels: "errorcheck"

sqs_url

Required

Enter the full URL for the SQS queue.

Review the following example:

sqs_url: "https://sqs.us-west-2.amazonaws.com/233765244907/taylan-test-sqs"

access_key_id

Required

Enter the access key ID that the agent will use to pull SQS and download S3 files. 

This access key ID must have the following permissions: 

  • sqs:DeleteMessage
  • sqs:DeleteMessageBatch
  • sqsReceiveMessage
  • s3:GetObject

Review the following example:

access_key_id: "ABCDEFG"

access_secret

Required

Enter the access secret key that corresponds with the access key ID.

Review the following example:

access_secret: "Dn2djaskl"

role_arn

Optional

Enter the ARN that has permissions to use the desired IAM Role

To learn more, review this document from AWS.

Review the following example:

role_arn: "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>"

external_id

Optional

Enter the external ID associated with the desired IAM role. 

To learn more, review this document from AWS.

Review the following example:

external_id: "053cf606-8e80-47bf-b849-8cd1cc826cfc"

region

Required

Enter the region where the bucket and SQS queue are located. 

Review the following example:

region: "us-west-2"

log_type

Optional

Enter the desired log types to monitor: 

  • alb
  • cloudtrail

Review the following example:

log_type: alb

filters

Optional

Enter an existing filter to add to this input. 

To learn how to create a filter, see Filters.

Review the following example:

filters: 
  - info

 

Share this document