Documentation
Cloud
AWS
Analytics
Kinesis

Kinesis

Amazon Kinesis Data Streams 筆記。

Intro

Kinesis Data Streams is a scalable and durable real-time data streaming service that can continuously capture gigabytes of data per second from hundreds of thousands of sources.

Setup Kinesis Agent On EC2

Create IAM Role

Attach on EC2:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData",
                "kinesis:PutRecords"
            ],
            "Resource": "*"
        }
    ]
}

Install Agent 

sudo yum update
sudo yum install –y aws-kinesis-agent

Create a folder for streaming data 

cd /opt/
sudo mkdir app-stream-1
cd app-stream-1

Config 

sudo cat /etc/aws-kinesis/agent.json

Default configuration:

{
  "cloudwatch.emitMetrics": true,
  "kinesis.endpoint": "https://kinesis.ap-southeast-1.amazonaws.com",
  "firehose.endpoint": "",

  "flows": [
    {
      "filePattern": "/opt/app-stream-1/logfile.log",
      "kinesisStream": "kds-test",
      "partitionKeyOption": "RANDOM"
    },
    {
      "dataProcessingOptions": {
        "optionName": "CSVTOJSON",
        "customFieldNames": [ "field1", "field2" ],
        "delimiter": "yourdelimiter"
      },
      "filePattern": "/tmp/app.log*",
      "kinesisStream": "stream name",
      "initialPosition": "END_OF_FILE",
      "maxBufferAgeMillis": "60000",
      "maxBufferSizeBytes": "4194304",
      "maxBufferSizeRecords": "500",
      "minTimeBetweenFilePollsMillis": "100",
      "multiLineStartPattern": "",
      "partitionKeyOption": "RANDOM",
      "skipHeaderLines": "0",
      "truncatedRecordTerminator": "\n"
    }
  ]
}

Set permissions 

sudo chown aws-kinesis-agent-user:aws-kinesis-agent-user -R /opt/stream-1

Kinesis Agent Services 

sudo service aws-kinesis-agent status
sudo service aws-kinesis-agent start
sudo service aws-kinesis-agent restart
sudo service aws-kinesis-agent stop

Tail log files

Tail the application log file:

sudo tail -f /opt/kds/file.log

Tail the Kinesis Agent log file:

sudo tail -f /var/log/aws-kinesis-agent/aws-kinesis-agent.log
GlueRedshift