JarCasting
Hopsworks CLI
A command-line client, written in Java, that uses the Hopsworks REST API for Job execution and file uploading.
Maven dependency
<dependency>
<groupId>io.hops</groupId>
<artifactId>hopsworks-cli</artifactId>
<version>0.1</version>
</dependency>
Configuration
You can either edit the conf/hops.properties file or set environment variables: HOPSWORKS_USER HOPSWORKS_PASSWORD HOPSWORKS_ENDPOINT=http://:/hopsworks HOPSWORKS_PROJECT=projectname
Command-line Utility
For help on using the command-line utility: ./hops-cli --help
Usage as a Library
The output library is a shaded jar file. You can include it
Running a job with hopsworks-cli
You need to follow a number of steps to run a job. First, you need to get the projectId from the project name, then copy your .jar or .py file into the project, then create a Job configuration and create the Job (if it doesn't already exist) in Hopsworks, then run the job, and finally download the job logs.
-
ProjectService. @Path("/getProjectInfo/{projectName}") name -> project_id Use ProjectService REST API
-
Copy filename to the /hopsworks-api/api//dataset/upload/ Use DatasetService REST API
-
Run Job with JobServer REST API Return application_attempt_id and jobId
-
Download Logs with application_attempt_id
Acknowledgement
This work was developed as part of the Streamline EU Project and it builds on earlier work done in the Aegis project: https://github.com/aegisbigdata/hopsworks-data-connector https://www.aegis-bigdata.eu/
