JarCasting
JarCasting JarCasting

spark-pipeline-driver

Spark Pipeline Driver Engine

Лицензия

Лицензия
Группа

Группа

com.acxiom
Идентификатор

Идентификатор

spark-pipeline-driver
Последняя версия

Последняя версия

1.4.0
Дата

Дата
Тип

Тип

pom
Описание

Описание

spark-pipeline-driver
Spark Pipeline Driver Engine
Ссылка на сайт

Ссылка на сайт

https://github.com/Acxiom/spark-pipeline-driver
Система контроля версий

Система контроля версий

https://github.com/Acxiom/spark-pipeline-driver

Скачать spark-pipeline-driver

Имя Файла Размер
spark-pipeline-driver-1.4.0.pom 13 KB
Обзор

Как подключить последнюю версию

<!-- https://jarcasting.com/artifacts/com.acxiom/spark-pipeline-driver/ -->
<dependency>
    <groupId>com.acxiom</groupId>
    <artifactId>spark-pipeline-driver</artifactId>
    <version>1.4.0</version>
    <type>pom</type>
</dependency>
// https://jarcasting.com/artifacts/com.acxiom/spark-pipeline-driver/
implementation 'com.acxiom:spark-pipeline-driver:1.4.0'
// https://jarcasting.com/artifacts/com.acxiom/spark-pipeline-driver/
implementation ("com.acxiom:spark-pipeline-driver:1.4.0")
'com.acxiom:spark-pipeline-driver:pom:1.4.0'
<dependency org="com.acxiom" name="spark-pipeline-driver" rev="1.4.0">
  <artifact name="spark-pipeline-driver" type="pom" />
</dependency>
@Grapes(
@Grab(group='com.acxiom', module='spark-pipeline-driver', version='1.4.0')
)
libraryDependencies += "com.acxiom" % "spark-pipeline-driver" % "1.4.0"
[com.acxiom/spark-pipeline-driver "1.4.0"]

Зависимости

compile (3)

Идентификатор библиотеки Тип Версия
org.json4s : json4s-native_2.11 jar 3.2.11
org.json4s : json4s-ext_2.11 jar 3.2.11
com.jcraft : jsch jar 0.1.55

provided (4)

Идентификатор библиотеки Тип Версия
org.scala-lang : scala-library jar 2.11.12
org.scala-lang : scala-compiler jar 2.11.12
org.apache.spark : spark-core_2.11 jar 2.3.3
org.apache.spark : spark-sql_2.11 jar 2.3.3

test (4)

Идентификатор библиотеки Тип Версия
junit : junit jar 4.11
org.scalatest : scalatest_2.11 jar 3.0.8
org.scalamock : scalamock-scalatest-support_2.11 jar 3.6.0
org.apache.hadoop : hadoop-minicluster jar 3.2.0

Модули Проекта

  • spark-pipeline-engine
  • common-pipeline-steps
  • streaming-pipeline-drivers
  • pipeline-drivers-examples
  • application-utils
Branch Build Coverage Quality
Develop Develop Build Develop Coverage Quality Gate Status
Master Master Status Master Coverage Quality Gate Status

Metalus Pipeline Library

The Metalus library was created as a way to create Spark applications at runtime without the need to write or compile code. The library is written in Scala and provides binaries for different version of Spark and Scala. Developers build applications by providing a JSON configuration file which gets loaded and executed by the metalus core library.

Documentation

Documentation for this project may be found here.

Contributing

Instructions for contributing to this project and instructions on building may be found here.

Projects

There are several sub-projects:

Metalus Pipeline Core

This project contains the core library and is the minimum requirement for any application.

Maven 2.11 Spark 2.3 library

Maven 2.11 Spark 2.4 library

Maven 2.12 Spark 2.4 library

Maven 2.12 Spark 3.0 library

Metalus Common Step Library

This step library contains steps that are considered generic enough to be used in any project.

Maven 2.11 Spark 2.3 library

Maven 2.11 Spark 2.4 library

Maven 2.12 Spark 2.4 library

Maven 2.12 Spark 3.0 library

Metalus AWS Step Library

This step library contains AWS specific components. The Kinesis driver provides a basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.

Maven 2.11 Spark 2.3 library

Maven 2.11 Spark 2.4 library

Maven 2.12 Spark 2.4 library

Maven 2.12 Spark 3.0 library

Metalus GCP Step Library

This step library contains GCP specific components. The Pub/Sub driver provides a basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.

Maven 2.11 Spark 2.3 library

Maven 2.11 Spark 2.4 library

Maven 2.12 Spark 2.4 library

Maven 2.12 Spark 3.0 library

Metalus Kafka Step Library

This step library contains GCP specific components. The Kafka driver provides a basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.

Maven 2.11 Spark 2.3 library

Maven 2.11 Spark 2.4 library

Maven 2.12 Spark 2.4 library

Maven 2.12 Spark 3.0 library

Metalus Mongo Step Library

This step library adds support for working with Mongo.

Maven 2.11 Spark 2.3 library

Maven 2.11 Spark 2.4 library

Maven 2.12 Spark 2.4 library

Maven 2.12 Spark 3.0 library

Metalus Pipeline Examples

This project provides several examples to help demonstrate how to use the library.

Metalus Utilities

This project provides utilities that help work with the project.

Maven 2.11 Spark 2.3 library

Maven 2.11 Spark 2.4 library

Maven 2.12 Spark 2.4 library

Maven 2.12 Spark 3.0 library

Metalus Application

This project provides a single jar that can be used to run the application. Additional components provide jars that can be added to the classpath.

Examples

Examples of building pipelines can be found in the metalus-examples project.

com.acxiom

Версии библиотеки

Версия
1.4.0
1.3.3
1.3.2
1.3.1
1.3.0
1.2.1