问题在windows 环境使用Java下调试远程虚拟机中的Hadoop集群报错，问题很奇怪，说是少了 winutils.exe 文件，而且少了HADOOPHOME 的环境变量；我是部署在虚拟机CentOS 7 上的集群，难道Windows 上使用它的Hadoop还需要自己安装环境，事实上，是真的。. But it is better to have it in place to avoid unnecessary problems. In order to avoid error, download winutils.exe binary and add the same to the classpath. Error: java.io.IOException: Could not locate executable null bin winutils.exe in the Hadoop binaries. Sample Code: The following program counts the number of lines containing the character. May 28, 2020 Copy the winutils.exe file from the Downloads folder to C:hadoopbin. Step 7: Configure Environment Variables Configuring environment variables in Windows adds the Spark and Hadoop locations to your system PATH.

How To Repair The Missing Winutils.exe Error In The Big Data ...
Winutils.exe Hadoop 2.7 Download

I solved this and it worked for me, hope it would help. It's a very small mistake I made - I saved the winutils executable as 'winutils.exe' instead of just winutils. So when the variable gets resolved it's been resolving to winutils.exe.exe which is nowhere in the Hadoop binaries. I removed that '.exe' and triggered the shell, it worked. Winutils.exe -help Usage: hadoop bin winutils.exe command. Provide basic command line utilities for Hadoop on Windows. The available commands and their usages are: chmod Change file mode bits. Usage: chmod OPTION OCTAL-MODE FILE or: chmod OPTION MODE FILE Change the mode of the FILE to MODE.

This article shows how to install a standalone Spark on Windows operating system.

Do These Steps
Clearing the Startup Hurdles

1. Prerequisites

Must-haves

Windows PowerShell
— It should have come with any Windows 7 or newer/Windows Server 2008 R2 or newer. If you want to upgrade your PowerShell (highly recommended), go here.
A program that can open & extract .tgz files, such as 7-zip , Winzip or Winrar.
Internet connection.

Nice-to-have

You need to do the following things to make Spark work (in standalone mode) on Windows:

Install Java Development Kit (JDK)
Set system environment variable JAVA_HOME
Download & extract Apache Spark
Set system environment variable SPARK_HOME
Download winutils.exe, a Hadoop file
Modify PATH environment variable so Windows can find Spark and winutils.exe

These steps are detailed below.

Step 1: Install Java JDK (Java development kit)

Google “Java JDK download”, go to Oracle’s website and download Java JDK. Don’t worry about JRE (Java runtime environment), because it is included as a part of JDK.

The download site may have different installers for 32-bit (x86) vs 64-bit (x64) operating systems. Use the following PowerShell command to find out what your system is:

It is highly recommended (though not absolutely necessary) to install Java JDK in a path that contains no spaces, for example C:Javajdk1.8.0_181. Something like C:Program FilesJavajdk1.8.0_181 is okay, but it brings some extra work to do later on.

To verify Java JDK has been successfully installed, restart (close & re-open) PowerShell, and run these commands:

You should see something like this:

Step 2: Set JAVA_HOME environment variable

You need to set JAVA_HOME environment variable to the directory where you installed Java JDK. Spark depends on this to find Java. It is IMPORTANT to make sure the value of JAVA_HOME contains no spaces, otherwise Spark won’t be happy.

If you chose to install JDK in a path that contains a space, e.g. like C:Program FilesJavajdk1.8.0_181, you need an extra step to convert that path to a 8.3 short filename, like C:PROGRA~1Javajdk1.8.0_181. Use the following PowerShell command. Be sure to replace C:Program FilesJavajdk1.8.0_181 with your own JDK installation directory.

If your JDK directory path contains no spaces, e.g. likeC:Javajdk1.8.0_181, use the following PowerShell command. Be sure to replace C:Javajdk1.8.0_181 with your own JDK installation directory.

More advancedly, if you have Administrator privilege (running PowerShell as Administrator), and know what you’re doing, use this command to set JAVA_HOME for the machine, instead of just for current user.

How To Repair The Missing Winutils.exe Error In The Big Data ...

Explanation: SetEnvironmentVariable() is a .NET function, reference here. The last argument (User/Machine) specifies scope of the environment variable, that is, whether to make it available for the current user only or for all users of this machine.

Step 3: Download & extract Apache Spark

Go to Apache Spark download page, and download Spark.

When download is complete, extract the content, which should be a folder named likespark-2.3.1-bin-hadoop2.7, and put that folder anywhere you like. I assume it’s placed at C:spark-2.3.1-bin-hadoop2.7 in the following steps.

Step 4: Set system environment variable SPARK_HOME

Similar to setting JAVA_HOME, set SPARK_HOME with the following PowerShell command. Be sure to replace C:spark-2.3.1-bin-hadoop2.7with your Spark directory.

Step 5: Download Hadoop’s winutils.exe

Go to Steve Loughran’s winutils GitHub repository, and download winutils.exe of your choice of Hadoop version. If you don’t want to face this choice, the latest Hadoop version should be OK. Save winutils.exe to anywhere you like, and remember where you put it.

Or, you can clone the entire repository (thus all versions of winutils.exe) to your computer:

This way I would find winutils at C:winutilshadoop-3.0.0binwinutils.exe

Explanation: winutils.exe is a part of Hadoop. It’s produced when one compiles Hadoop from source code on Windows, which can be a painful process. Fortunately, Steve has shared with the world the winutils.exe files he compiled, so we don’t have to know how to compile Hadoop on Windows.

Step 6: Modify PATH environment variable so Windows can find Spark and winutils.exe

Now we need to let Windows know where to find Spark and Hadoop winutils.exe. Again, I assume they are placed at the following paths, you need to change them accordingly depending on where you put them.

C:spark-2.3.1-bin-hadoop2.7
C:winutilshadoop-3.0.0binwinutils.exe

Run these commands in PowerShell:

(Again, you may replace ‘User’ with ‘Machine’ if you know what you’re doing.)

When these are complete, close and re-open PowerShell.

Explanation: The directory name where spark files are copied should include any space.

Run the following PowerShell commands line by line. Each line should return a True to you. If you receive anything other than True, something is wrong.

At this point, you should be able to launch an interactive Spark shell, either in PowerShell or Command Prompt, with spark-shell (Scala shell), pyspark (Python shell), or sparkR (R shell).

However, it is not that straightforward, and you will face various issues as listed below, along with their resolutions.

Please note that you must have administrative permission to the user or you need to run command tool as administrator.

Issue 1: Failed to Locate winutils Binary

Even if you don’t use Hadoop, Windows needs Hadoop to initialize the “hive” context. You get the following error if Hadoop is not installed

This can be fixed by adding a dummy Hadoop installation. Spark expects winutils.exe in the Hadoop installation “<Hadoop Installation Directory>/bin/winutils.exe” (note the “bin” folder).

Download Hadoop 2.7’s winutils.exe and place it in a directory C:InstallationsHadoopbin
Now set HADOOP_HOME = C:InstallationsHadoop environment variables.

Now start the Windows shell; you may get few warnings, which you may ignore for now

1. Prerequisites

Must-haves

Nice-to-have

How To Repair The Missing Winutils.exe Error In The Big Data ...

Issue 1: Failed to Locate winutils Binary

Issue 2: File Permission Issue for /t

Winutils Exe Hadoop