A new Spark Web UI: Spark App

Hi Spark Makers!


A Hue Spark application was recently created. It lets users execute and monitor Spark jobs directly from their browser and be more productive.


We previously released the app with an Oozie submission backend but switched to the Spark Job Server (SPARK-818) contributed by Ooyala and Evan’s team at the last Spark Summit. This new server will enable a real interactivity with Spark and is closer to the community.

 

We hope to work with the community and have support for Python, Java, direct script submission without compiling/uploading and other improvements in the future!


As usual feel free to comment on the hue-user list or @gethue! About questions directly related to Job Server, participate on the pull request, SPARK-818 or the Spark user list!


Get Started!

Currently only Scala jobs are supported and programs need to implement this trait and be packaged into a jar. Here is a WordCount example. To learn more about Spark Job Server, check its README.


Requirements

We assume you have Scala installed on your system.


Get Spark Job Server

Currently on github on this branch:

git clone https://github.com/ooyala/incubator-spark.git spark-server
cd spark-server
git checkout -b jobserver-preview-2013-12 origin/jobserver-preview-2013-12


Then type:

sbt/sbt
project jobserver
re-start


Get Hue

Currently only on github (will be in CDH5b2):

https://github.com/cloudera/hue#getting-started


If Hue and Spark Job Server are not on the same machine update the hue.ini property in desktop/conf/pseudo-distributed.ini:

[spark]
  # URL of the Spark Job Server.
  server_url=http://localhost:8090/

Get a Spark example to run

Then follow this walk-through and create the example jar that is used in the video demo.


This article was originally posted 7 months ago.

Tags: video tutorial spark


comments powered by Disqus

Blog Archive

Browse archive

Blog Tags

loading...