JarCasting
JarCasting JarCasting

Amazon Kinesis Storm Spout for Java

The Amazon Kinesis Storm Spout helps Java developers integrate Amazon Kinesis with Storm.

License

License
Categories

Categories

AWS Container PaaS Providers ORM Data
GroupId

GroupId

com.amazonaws
ArtifactId

ArtifactId

kinesis-storm-spout
Last Version

Last Version

1.1.1
Release Date

Release Date
Type

Type

jar
Description

Description

Amazon Kinesis Storm Spout for Java
The Amazon Kinesis Storm Spout helps Java developers integrate Amazon Kinesis with Storm.
Project URL

Project URL

https://aws.amazon.com/kinesis
Source Code Management

Source Code Management

https://github.com/awslabs/kinesis-storm-spout.git

Download kinesis-storm-spout

Filename Size
kinesis-storm-spout-1.1.1.pom
kinesis-storm-spout-1.1.1.jar 59 KB
kinesis-storm-spout-1.1.1-sources.jar 46 KB
kinesis-storm-spout-1.1.1-javadoc.jar 92 KB
Browse

How to add to project

<!-- https://jarcasting.com/artifacts/com.amazonaws/kinesis-storm-spout/ -->
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>kinesis-storm-spout</artifactId>
    <version>1.1.1</version>
</dependency>
// https://jarcasting.com/artifacts/com.amazonaws/kinesis-storm-spout/
implementation 'com.amazonaws:kinesis-storm-spout:1.1.1'
// https://jarcasting.com/artifacts/com.amazonaws/kinesis-storm-spout/
implementation ("com.amazonaws:kinesis-storm-spout:1.1.1")
'com.amazonaws:kinesis-storm-spout:jar:1.1.1'
<dependency org="com.amazonaws" name="kinesis-storm-spout" rev="1.1.1">
  <artifact name="kinesis-storm-spout" type="jar" />
</dependency>
@Grapes(
@Grab(group='com.amazonaws', module='kinesis-storm-spout', version='1.1.1')
)
libraryDependencies += "com.amazonaws" % "kinesis-storm-spout" % "1.1.1"
[com.amazonaws/kinesis-storm-spout "1.1.1"]

Dependencies

compile (4)

Group / Artifact Type Version
com.amazonaws : aws-java-sdk jar 1.7.13
org.apache.commons : commons-lang3 jar 3.0
org.apache.storm : storm-core jar 0.9.2-incubating
com.netflix.curator : curator-framework jar 1.1.3

provided (1)

Group / Artifact Type Version
com.google.guava : guava jar 13.0-final

Project Modules

There are no modules declared in this project.

Amazon Kinesis Storm Spout

The Amazon Kinesis Storm spout helps Java developers integrate Amazon Kinesis with Storm.

Requirements

Overview

The Amazon Kinesis Storm spout fetches data records from Amazon Kinesis and emits them as tuples. The spout stores checkpoint state in ZooKeeper to track the current position in the stream.

The Amazon Kinesis Storm spout can be configured to retry failed records. By default, it retries a failed record 3 times. If a record fails and the retry limit has been reached, the spout will log an error and skip over the record. The spout buffers pending records in memory, so it can re-emit a failed record without having to re-fetch the record from Amazon Kinesis. The spout sets the checkpoint to the highest sequence number that has been ack'ed (or exhausted retry attempts).

To use the spout, you'll need to add it to your Storm topology.

  • KinesisSpout: Constructs an instance of the spout, using your AWS credentials and the configuration specified in KinesisSpoutConfig (as well as com.amazonaws.ClientConfiguration, via the AWS SDK). Each task executed by the spout operates on a distinct set of Amazon Kinesis shards. Shard states are periodically committed to ZooKeeper. When the spout is deactivated, it will disconnect from ZooKeeper, but the spout will continue monitoring its local state so you can activate it again later.
  • KinesisSpoutConfig: Configures the spout, including the Storm topology name, the Amazon Kinesis stream name, the endpoint for connecting to ZooKeeper, and the prefix for the ZooKeeper paths where the spout state is stored. See the samples folder for configuration examples.
  • DefaultKinesisRecordScheme: This default scheme, used by the sample topology, emits a tuple of (partitionKey, record). If you want to emit more structured data, you can provide your own implementation of IKinesisRecordScheme.

The samples folder includes a sample topology and sample bolt, using the number of Amazon Kinesis shards as the parallelism hint for the spout. For more information about Storm topologies and bolts, see the Storm documentation.

Using the Sample

  1. Edit the *.properties file to configure your Storm topology, Amazon Kinesis stream, and ZooKeeper details. For your AWS Credentials, we recommend using IAM roles on Amazon EC2 when possible. You can also specify your credentials using system properties, environment variables, or AwsCredentials.properties.
  2. Package the spout and the sample (including all dependencies but excluding Storm itself) into one JAR file.
  3. Deploy the package to Storm via the JAR file, e.g., storm jar my-spout-sample.jar SampleTopology sample.properties RemoteMode

Release Notes

Release 1.1.1 (June 1, 2015)

  • When a Kinesis Stream is resharded, "storm rebalance" can be invoked to refresh the shard list and distribute the latest shards across the Spout tasks.

Release 1.1.0 (October 21, 2014)

  • Added support for retrying failed records
  • Added region name support

Future Work

  • Automatically handle closed, split, and merged shards.

Related Resources

Amazon Kinesis Developer Guide
Amazon Kinesis API Reference

Amazon Kinesis Client Library
Amazon Kinesis Connector Library

com.amazonaws

Amazon Web Services - Labs

AWS Labs

Versions

Version
1.1.1
1.1.0