JarCastingDataMover
Out of the box scheduling, logging, monitoring and data governance for your scala ETL jobs.
Getting DataMover
<dependency>
<groupId>de.jannikarndt</groupId>
<artifactId>datamover_2.12</artifactId>
<version>1.2.0</version>
</dependency>
or
libraryDependencies += "de.jannikarndt" % "datamover" % "1.1.0"
Example:
import de.jannikarndt.datamover._
import de.jannikarndt.datamover.governance.GovernedID
import scala.concurrent.duration._
import scala.language.postfixOps
object ExampleJob {
def main(args: Array[String]): Unit = DataMover run classOf[ExampleJob] every (10 seconds)
}
class ExampleJob extends DataMover("ExampleJob") {
override def run(): Unit = {
// here you can access
// - logger => Log debug, info or error information
// - monitor => track throughput
// - governedId => append this to your output to find the job that generated it
// Logging
logger.info(s"Logs are aggregated per run. These are for Job ${governedId.identifier}.")
// Write you own EXTRACT-function
// Monitor your input
monitor.input(5)
// Write your own TRANSFORM-function
// Write your own LOAD-function
// Monitor your output
monitor.output("Appended successfully")
}
}
Then head to http://localhost:55555 (and increasing for every additional job):
Monitoring with Prometheus
Prometheus can read directly from DataMover jobs. Just add the targets: ['localhost:55555'] to your prometheus.yml!
License
This code is open source software licensed under the MIT License.
To-Do / Planned
-
Data Governance
-
Alerting when job fails
-
Uptime-Monitoring for sources and sinks
-
Versioning: Which version of which job is deployed where? + Changelog
-
Feature toggles
-
Central server to monitor all jobs:
-
Interface for Elastic/Kibana
-
Interface for Jolokia
Deployment
Snapshots are deployed at oss.sonatype.org. Releases are deployed at maven.org.
Changes
v1.3.0
- Support for Prometheus
- Jobs automatically choose a free port, starting at 55555
- LogLevel coloring
- GovernedId can be accessed anywhere in class
v1.2.0
- Logger now supports ERROR, WARN and DEBUG
- Governor writes valid json
- artifact id contains scala version
v1.1.0
- Upgrade to scala 2.12.3
- Removed unnecessary dependencies
- Removed old files