spark_1.x_submit_examples

------------------------------------------------------------------------------------------------
SPARK SUBMIT EXAMPLES
------------------------------------------------------------------------------------------------

Build Commands:
--------------------------------------------------------------

cd /home/orienit/spark/workspace/KalyanSparkProjectMaven
mvn clean eclipse:eclipse    (to prepare eclipse project)
mvn package (to prepare jar file)

cd /home/orienit/spark/workspace/KalyanSparkProjectSbt
sbt clean eclipse    (to prepare eclipse project)
sbt package (to prepare jar file)


--------------------------------------------------------------


$SPARK_HOME/bin/run-example SparkPi 10
$SPARK_HOME/bin/run-example JavaWordCount file:///home/orienit/spark/input/demoinput


$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master local $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.JavaWordCount --master local $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar file:///home/orienit/spark/input/demoinput

$SPARK_HOME/bin/spark-submit $SPARK_HOME/examples/src/main/python/pi.py

$SPARK_HOME/bin/spark-submit $SPARK_HOME/examples/src/main/python/wordcount.py file:///home/orienit/work/input/demoinput

$SPARK_HOME/bin/spark-submit $SPARK_HOME/examples/src/main/r/dataframe.R


Execute in Local Mode:
===============================
$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master local $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master local[4] $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master local[*] $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10


Start Command for Spark:
-----------------------------
$SPARK_HOME/sbin/start-all.sh


Start Command for Spark:
-----------------------------
$SPARK_HOME/sbin/stop-all.sh



Execute in Cluster Mode:
===============================
$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://kalyan:7077 $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://kalyan:7077 --deploy-mode client $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://kalyan:7077 --deploy-mode cluster $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10



Python example:
===================
$SPARK_HOME/bin/spark-submit --master spark://kalyan:7077 $SPARK_HOME/examples/src/main/python/pi.py 10



Yarn example:
=====================
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10


Note:
1. --master yarn-client
(or)
   --master yarn --deploy-mode client

2. --master yarn-cluster
(or)
   --master yarn --deploy-mode cluster





Own examples:
======================
$SPARK_HOME/bin/spark-submit --class <main class of the job> --master <master> <path of the jar file> <input> <output>



$SPARK_HOME/bin/spark-submit --class com.orienit.spark.scala.training.WordCount --master local /home/orienit/spark/workspace-july-2016/KalyanSparkProjectSbt/target/scala-2.11/kalyansparkprojectsbt_2.11-1.0.jar file:///home/orienit/spark/input/demoinput file:///home/orienit/spark/output/demooutput


$SPARK_HOME/bin/spark-submit --class com.orienit.spark.training.scala.WordCount --master local /home/orienit/spark/workspace-jan-16/SparkSbtProject1/target/scala-2.11/spark-project_2.11-1.0.jar file:///home/orienit/spark/input/demoinput file:///home/orienit/spark/output/demooutput-custom


$SPARK_HOME/bin/spark-submit --class com.orienit.spark.training.WordCount --master local /home/orienit/spark/workspace/KalyanSparkProjectMaven/target/KalyanSparkProjectMaven-0.0.1-SNAPSHOT.jar file:///home/orienit/spark/input/demoinput

$SPARK_HOME/bin/spark-submit --class com.orienit.spark.training.WordCount --master local /home/orienit/spark/workspace/KalyanSparkProjectSbt/target/scala-2.11/kalyansparkprojectsbt_2.11-1.0.jar file:///home/orienit/spark/input/demoinput