Installing Apache Spark Standalone-Cluster in Windows

Here I will try to elaborate on simple guide to install Apache Spark on Windows (Without HDFS) and link it to local standalong Hadoop Cluster. Local hadoop cluster can be invoked by start-all.cmd on command prompt and viewed at http://localhost:50070. If it’s not installed follow here. Also read http://sachingpta.gitlab.io/sparkintro/ to resolve many issues.

<hr> **Table of Contents** [TOCM] <hr> # Prerequisits for Apache Spark - **JAVA**: Install [Java SE Development Kit (x64)](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html) and set `C:\PROGRA~1\Java\jdk1.8.0_131` as `JAVA_HOME` enviornment variable and append `%JAVA_HOME%\bin` to `PATH`. You should be able to see java on console by executing `java -version`. Output for my system is as shown ``` C:\>java -version java version "1.8.0_131" Java(TM) SE Runtime Environment (build 1.8.0_131-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode) ``` - **Maven**: Get [Apache Bniary](https://maven.apache.org/download.cgi) and extract [apache-maven-3.5.0-bin.tar.gz](http://mirror.fibergrid.in/apache/maven/maven-3/3.5.0/binaries/apache-maven-3.5.0-bin.tar.gz) to `C:\APPs\ApacheSuite\Maven`. Setup two enviornment variables `M2_HOME` and `MAVEN_HOME` with value `C:\APPs\ApacheSuite\Maven` and append to `PATH` value `%M2_HOME\bin`. You should be able to execute `mvn --version` from command prompt. Its output is shown below ``` C:\>mvn --version C:\ Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-04T01:09:06+05:30) Maven home: C:\APPs\ApacheSuite\Maven\bin\.. Java version: 1.8.0_131, vendor: Oracle Corporation Java home: C:\PROGRA~1\Java\jdk1.8.0_131\jre Default locale: en_US, platform encoding: Cp1252 OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows" ``` - **SBT (Scala Build Tools)**: Install from [MSI Setup](http://www.scala-sbt.org/download.html) in location `C:\APPs\sbt`. Setup `SBT_HOME` enviornment vairbale to `C:\APPs\sbt\` and appened `%SBT_HOME%\bin` to `PATH`. Then use `sbt sbtVersion` to check version of SBT. Output shall be as follows: ``` C:\>sbt sbtVersion Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0 [warn] The global sbt directory is now versioned and is located at C:\Users\Admin\.sbt\0.13. [warn] You are seeing this warning because there is global configuration in C:\Users\Admin\.sbt but not in C:\Users\Admin\.sbt\0.13. [warn] The global sbt directory may be changed via the sbt.global.base system property. [info] Set current project to root (in build file:/C:/) [info] 0.13.13 ``` - **SCALA (Scala Programming Language)**: Install from [MSI Setup](https://www.scala-lang.org/download/) in location `C:\APPs\Scala`. Setup `SCALA_HOME` enviornment vairbale to `C:\APPs\Scala` and appened `%SCALA_HOME%\bin` to `PATH`. Then use `scala -version` to check version of SBT. Output shall be as follows: ``` C:\>scala -version Scala code runner version 2.12.1 -- Copyright 2002-2016, LAMP/EPFL and Lightbend, Inc. ``` - **Optional Items**: - [Scala IDE](http://scala-ide.org/download/sdk.html): Includes ` Eclipse 4.6.1 (Neon)`, `Scala IDE 4.5.0`, `Scala 2.11.8 and Scala 2.10.6` with other stuff. It's a complete ecosystem for edit, debug and work with **_Scala Projects (or Spark Projects_**). You can also install [Eclipse Neon](http://www.eclipse.org/neon/) and add `Scala IDE` from Eclipse Marketplace (_This requires you to have SCALA installed and not a recommended practice_) - [SBT Eclipse](https://github.com/typesafehub/sbteclipse): If you've created a Scala project using notepad and compiling it using `sbt` from command line. You can upgrade this project to be used in Eclipse by means of this cool [SBT Eclipse Plugin](https://github.com/typesafehub/sbteclipse). Instructions to use this are availaible [here](https://sachingpta.gitlab.io/_posts/apache-spark-getting-started.html) and in detailed [guide here](http://scalatutorials.com/beginner/2013/07/18/getting-started-with-sbt/) # Install Local{*} Cluster ## 1. Install Local HDFS Cluster This installation will link to Hadoop HDFS cluster on seperate system. Local hadoop cluster can be invoked by `start-all.cmd` on command prompt and viewed at [http://localhost:50070](http://localhost:50070). If it's not installed follow [here](http://sachingpta.gitlab.io/hadoop/) to install Hadoop Suite (Yarn, HDFS, MapReduce First). **_Note:_** In order to link to existing Hadoop make sure that enviornment variable `HADOOP_HOME` points location of parent folder of Hadoop Binary (**_WinUtils.exe_**). That means `%HADOOP_HOME% = C:\APPs\ApacheSuite\Hadoop` ## 2. Install Apache Spark On Windows Download [Apache Spark](http://spark.apache.org/downloads.html) version titled **_pre built with user provided hadoop_** as file [spark-2.1.1-bin-without-hadoop.tgz](https://d3kbcqa49mib13.cloudfront.net/spark-2.1.1-bin-without-hadoop.tgz). Extract this file to `C:\APPs\ApacheSuite\Spark` folder. Set `SPARK_HOME` enviornment variable to point `C:\APPs\ApacheSuite\Spark` and append `%SPARK_HOME%\bin` to `PATH` for access to `Spark Shell` ## 3. Fix Spark Batch File (spark-class2.cmd) **ERROR HANDLING** If you get following error on execution of `spark-shell`. Change line `set RUNNER="%JAVA_HOME%\bin\java"` in `bin\spark-class2.cmd` to `set RUNNER=%JAVA_HOME%\bin\java` on Windows. This is beacuse supplied official spark batch file is not properly written to execute `java.exe`. Use `CTRL+D` anytime to exit `spark-shell`. ``` C:\>spark-shell '""C:\Program' is not recognized as an internal or external command, operable program or batch file. ``` ## 4. Configure External HDFS & YARN Jars If you execute `spark-shell` with external HDFS you'll recieve following error, because this distribution of Spark is without HDFS and Spark now can't find HDFS/YARN classes. ``` Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream .... ... 7 more ``` This [StackOverflow](http://stackoverflow.com/questions/30906412/noclassdeffounderror-com-apache-hadoop-fs-fsdatainputstream-when-execute-spark-s) answer based on [Spark Official](https://spark.apache.org/docs/latest/hadoop-provided.html) for Linux, elaborated main steps on how to configure external HDFS & yarn. Which is as follows: Execute `%HADOOP_HOME%\bin\hadoop classpath` and copy entire output. Set an system enviornment variable `SPARK_DIST_CLASSPATH` with this value. On my system this is: _C:\APPs\ApacheSuite\Hadoop\etc\hadoop;C:\APPs\ApacheSuite\Hadoop\share\hadoop\common\lib\*;C:\APPs\ApacheSuite\Hadoop\share\hadoop\common\*;C:\APPs\ApacheSuite\Hadoop\share\hadoop\hdfs;C:\APPs\ApacheSuite\Hadoop\share\hadoop\hdfs\lib\*;C:\APPs\ApacheSuite\Hadoop\share\hadoop\hdfs\*;C:\APPs\ApacheSuite\Hadoop\share\hadoop\yarn\lib\*;C:\APPs\ApacheSuite\Hadoop\share\hadoop\yarn\*;C:\APPs\ApacheSuite\Hadoop\share\hadoop\mapreduce\lib\*;C:\APPs\ApacheSuite\Hadoop\share\hadoop\mapreduce\*_. Additionally create a file named `spark-env.cmd` in `/conf` folder of Spark. with following content and call this file before `spark-shell` everytime. ``` <a href="https://twitter.com/ECHO" class="user-mention">@ECHO</a> OFF set SPARK_DIST_CLASSPATH=<paste output="" here=""> ``` ## 5. Testing Word Counting With HDFS File 1. Start all Hadoop Services using `start-all.cmd` and test HDFS Web Access at: [http://localhost:50070/explorer.html#/](http://localhost:50070/explorer.html#/) 2. Use `hadoop fs -put README.md /` to upload _**README.md**_ file to root of HDFS. 3. Watch this file using `hadoop fs -ls /`or `hadoop dfs -ls /` or on Web UI at [http://localhost:50070/explorer.html#/](http://localhost:50070/explorer.html#/). 4. Now launch `spark-shell` and its user interface [http://localhost:4040](http://localhost:4040) and see file loaded to Hadoop ### Count Words of C:\APPs\ApacheSuite\Spark\README.md Use following code, **_note use of file:/// directive_** ``` val lines = sc.textFile("file:///C:/APPs/ApacheSuite/Spark/README.md") val words = lines.flatMap(_.split("\\s+")) val wc = words.map(w => (w, 1)).reduceByKey(_ + _) wc.foreach(println) wc.saveAsTextFile("file:///C:/APPs/ApacheSuite/Spark/output.txt") ``` ### Count Words of HDFS://README.md Just use `hdfs:` instead of `file:` as shown ``` val lines = sc.textFile("hdfs:///README.md") val words = lines.flatMap(_.split("\\s+")) val wc = words.map(w => (w, 1)).reduceByKey(_ + _) wc.foreach(println) wc.saveAsTextFile("file:///C:/APPs/ApacheSuite/Spark/output.txt") ``` Stop HDFS using `stop-all.cmd` and quit from `spark-shell` using `CTRL_D`. </textarea> </div> </paste>