安装hadoop
用root账号建立一个软件安装目录。
mkdir -p /appchown cloud.cloud /app将已下载好的hadoop安装包等通过putty套件中的psftp上传到已安装好的虚拟机 /app 目录。
题外话:psftp传输请用 open 192.168.10、 输入root、 输入 密码、lcd d:\xx 改变本地目录、cd /app 改变远程目录。put 文件名上传、quit退出。
方便的话可以下载securefx、xshell、secureCRT等工具完成。当然也可以直接用wget在linux中下载.
cd /apptar -zxvf hadoop-2.10.1.tar.gzln -s hadoop-2.10.1.tar.gz hadoopecho "master" >/app/hadoop/etc/hadoop/slaves #设定有集群服务器别名列表，由于是伪分布只填写本机器名称。- 配置core-site.xml
vi /app/hadoop/etc/hadoop/core-site.xml<configuration><property><name>fs.defaultFS</name><value>hdfs://master:8020</value></property><property> <name>fs.default.name</name> <value>hdfs://master:9000</value> </property><property><name>hadoop.tmp.dir</name><value>/tmp</value></property></configuration>- 配置hdfs-site.xml
<configuration><property><name>dfs.namenode.name.dir</name><value>/app/hadoop/hadoop/dfs/name</value></property><property> <name>dfs.datanode.data.dir</name><value>/app/hadoop/hadoop/dfs/data</value></property><property><name>dfs.replication</name><value>1</value></property></configuration>- 配置yarn-site.xml
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property><property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value></property> </configuration>- 配置队列名称 capacity-scheduler.xml 默认即可。
- 配置hadoop环境变量
vi ~/.bashrc
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdkexport JRE_HOME=$JAVA_HOME/jreexport CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATHexport PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATHHADOOP_HOME=/app/hadoopexport HADOOP_HOMEPATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATHexport PATH运行
. ~/.bashrc #使参数生效。- 格式化namenode
hadoop namenode -format注意：这个命令只需运行一次，每次运行它会生成一个新clusterID，从而造成集群ID和版本号不一致，从而不能正常启动。需要手动删除hadoop/dfs目录再重新格式化即可，当然也可手动修改id。
- 启动集群
start-dfs.shstart-yarn.sh--------------启动完成可用jps看到如下进程hadoop安装完成----------------------jps8769 SecondaryNameNode9096 NodeManager8444 NameNode8988 ResourceManager8589 DataNode9150 JpsHive安装
- 解压安装文件
tar -zxvf apache-hive-2.3.9-bin.tar.gzln -s apache-hive-2.3.9-bin hive- 配置环境变量 ~/.bashrc
export HIVE_HOME=/app/hiveexport PATH=$PATH:$HIVE_HOME/bin:$HIVE_HOME/confexport HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib" export HIVE_CONF_DIR=${HIVE_HOME}/conf export CLASS_PATH=.:${JAVA_HOME}/lib:${HIVE_HOME}/lib:$CLASS_PATHexport PATH=.:${HIVE_HOME}/bin:$PATHexport JAVA_LIBRARY_PATH=${HADOOP_HOME}/lib/native- 复制hive meta的mysql驱动
用cloud账号复制mysql驱动mysql-connector-java.jar到/app/hive/lib下
cp /app/mysql-connector-java.jar /app/hive/lib/- mysql安装
mysql用于存储hive元数据，该操作需要切换到root账号。准备好mysql-5.7.28-1.el7.x86_64.rpm-bundle.tar文件，笔者已经在网盘中准备好了该文件，请下载后上传到虚拟机中/app目录。
tar xvf mysql-5.7.28-1.el7.x86_64.rpm-bundle.tarrpm -ivh mysql-community-server-5.7.28-1.el7.x86_64.rpm mysql-community-client-5.7.28-1.el7.x86_64.rpm mysql-community-devel-5.7.28-1.el7.x86_64.rpm mysql-community-libs-* mysql-community-common-5.7.28-1.el7.x86_64.rpm mysql-community-embedded*如果安装过*程中报错，说明系统中包冲突。error: Failed dependencies:mariadb-libs is obsoleted by mysql-community-libs-5.7.28-1.el7.x86_64mariadb-libs is obsoleted by mysql-community-libs-compat-5.7.28-1.el7.x86_64只需要remove掉即可解决冲突----------------------------------------------------------------------yum remove mariadb-libs -ysystemctl start mysqld #启动mysqlsystemctl enable mysqld #设成开机自启配置/etc/my.cnf，完成后重启下systemctl restart mysqld
[client]default-character-set = utf8mb4 [mysql]default-character-set = utf8mb4[mysqld]datadir=/var/lib/mysqlsocket=/var/lib/mysql/mysql.sockcharacter-set-server = utf8mb4collation-server = utf8mb4_unicode_ciinit_connect='SET NAMES utf8mb4'validate_password=off #关闭复杂密码校验功能，生产环境请不要这样做symbolic-links=0log-error=/var/log/mysqld.logpid-file=/var/run/mysqld/mysqld.pid取到mysql系统 root 密码
grep "A temporary password" /var/log/mysqld.log--------------------------------------------------------- 2022-02-20T07:06:08.776350Z 1 [Note] A temporary password is generated for root@localhost: hylg#9;dlsSC 看到返回的内容，则初始的root密码为 hylg#9;dlsSC --------------------------------------------------------修改root密码：
mysqladmin -uroot -p'hylg#9;dlsSC' password 123456建立hive meta库和授权
create database hive;grant all privileges on hive.* to 'hive'@'%' identified by '123456';flush privileges;- 设置hive-site.xml
cd /app/hive/confcp /app/hive/conf/hive-default.xml.template /app/hive/conf/hive-site.xmlvi /app/hive/conf/hive-site.xml分别搜索/ConnectionPassword /ConnectionURL /ConnectionUserName /hive.exec.local.scratchdir /hive.downloaded.resources.dir 在其value项中填写我们前面设置的用户名、密码、连接数据库的参数、临时目录等。
<property> <name>hive.exec.local.scratchdir</name> <value>/tmp</value> <description>Local scratch space for Hive jobs</description> </property> <property> <name>hive.downloaded.resources.dir</name> <value>/tmp</value> <description>Temporary local directory for added resources in the remote file system.</description> </property><property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> <description>Username to use against metastore database</description> </property> <property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>123456</value> <description>password to use against metastore database</description> </property><property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://192.168.1.10:3306/hive?createDatabaseIfNotExist=true</value> <description> JDBC connect string for a JDBC metastore. To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL. For example, jdbc:postgresql://myhost/db?ssl=true for postgres database. </description></property>- 设置hive-env.sh
cp hive-env.sh.template hive-env.shvi hive-env.sh 增加以下参数：
export HADOOP_HOME=/app/hadoopexport HIVE_CONF_DIR=/app/hive/confexport HIVE_AUX_JARS_PATH=/app/hive/lib- 初始化hive meta信息表
schematool -dbType mysql -initSchema- 至此，恭喜各位hive已安装完成了，大家感兴趣还可以把spark和hbase安装上，这个以后我们再介绍。
[cloud@master conf]$ hive -e "show databases;"which: no hbase in (.:/app/hive/bin:/app/hadoop/bin:/app/hadoop/sbin:/usr/lib/jvm/java-1.8.0-openjdk/bin:/usr/lib/jvm/java-1.8.0-openjdk/jre/bin:/usr/lib/jvm/java-1.8.0-openjdk/bin:/usr/lib/jvm/java-1.8.0-openjdk/jre/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/app/hive/bin:/app/hive/conf)SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/app/apache-hive-2.3.9-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/app/hadoop-2.10.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]Logging initialized using configuration in jar:file:/app/apache-hive-2.3.9-bin/lib/hive-common-2.3.9.jar!/hive-log4j2.properties Async: trueOKdefaultTime taken: 5.277 seconds, Fetched: 1 row(s)- yarn监控地址： http://192.168.1.10:8088/cluster/
版权声明：内容来源于互联网和用户投稿 如有侵权请联系删除