Monday 23 September 2019

spark_projects_build_commands

How to Build Eclipse Project:
--------------------------------------------

1. Scala + Maven + Eclipse project
--------------------------------------------
cd /home/orienit/spark/workspace/KalyanScalaProjectMaven
mvn clean eclipse:eclipse
mvn package

scala -cp /home/orienit/spark/workspace/KalyanScalaProjectMaven/target/KalyanScalaProjectMaven-0.0.1-SNAPSHOT.jar com.orienit.scala.training.HelloWorld


2. Scala + Sbt + Eclipse project
--------------------------------------------
cd /home/orienit/spark/workspace/KalyanScalaProjectSbt
sbt clean eclipse
sbt package

scala -cp /home/orienit/spark/workspace/KalyanScalaProjectSbt/target/scala-2.11/kalyanscalaprojectsbt_2.11-1.0.jar com.orienit.scala.training.HelloWorld



3. Spark + Maven + Eclipse project
--------------------------------------------
cd /home/orienit/spark/workspace/KalyanSparkProjectMaven
mvn clean eclipse:eclipse
mvn package

Scala Code:
------------------
$SPARK_HOME/bin/spark-submit --class com.orienit.spark.scala.training.WordCount --master local /home/orienit/spark/workspace/KalyanSparkProjectMaven/target/KalyanSparkProjectMaven-0.0.1-SNAPSHOT.jar /etc/hosts /home/orienit/spark/ouput/maven-op-1

Java Code:
------------------
$SPARK_HOME/bin/spark-submit --class com.orienit.spark.java.training.WordCount --master local /home/orienit/spark/workspace/KalyanSparkProjectMaven/target/KalyanSparkProjectMaven-0.0.1-SNAPSHOT.jar /etc/hosts /home/orienit/spark/ouput/maven-op-2

Python Code:
------------------
$SPARK_HOME/bin/spark-submit /home/orienit/spark/workspace/KalyanSparkProjectMaven/src/main/python/com/orienit/spark/python/training/WordCount.py /etc/hosts /home/orienit/spark/ouput/maven-op-3


4. Spark + Sbt + Eclipse project
--------------------------------------------
cd /home/orienit/spark/workspace/KalyanSparkProjectSbt
sbt clean eclipse
sbt package


Scala Code:
------------------
$SPARK_HOME/bin/spark-submit --class com.orienit.spark.scala.training.WordCount --master local /home/orienit/spark/workspace/KalyanSparkProjectSbt/target/scala-2.11/kalyansparkprojectsbt_2.11-1.0.jar /etc/hosts /home/orienit/spark/ouput/sbt-op-1

Java Code:
------------------
$SPARK_HOME/bin/spark-submit --class com.orienit.spark.java.training.WordCount --master local /home/orienit/spark/workspace/KalyanSparkProjectSbt/target/scala-2.11/kalyansparkprojectsbt_2.11-1.0.jar /etc/hosts /home/orienit/spark/ouput/sbt-op-2

Python Code:
------------------
$SPARK_HOME/bin/spark-submit /home/orienit/spark/workspace/KalyanSparkProjectSbt/src/main/python/com/orienit/spark/python/training/WordCount.py /etc/hosts /home/orienit/spark/ouput/sbt-op-3








2 comments:

spark_streaming_examples

Create Spark Streaming Context: ========================================== scala: --------------- import org.apache.spark._ import ...