hadoop安装 hadoop+thrift+hive.docx

1、Hadoop + hive + thrift 安装文档一、主服务器设置1.创建用户#useradd hadoop2.设置密码#passwd hadoop3.切换 hadoop 登录#su - hadoop4.解压 hadoop 压缩包#tar zxvf hadoop-1.0.3.tar.gz5.设置目录权限#chown -R hadoop.hadoop hadoop-1.0.36.编辑环境变量#vim hadoop-1.0.3/conf/hadoop-env.sh编辑“JAVA_HOME”路径：export JAVA_HOME=/usr/local/jdk1.7.0_05增加一行（取消过时警告

2、）：export HADOOP_HOME_WARN_SUPPRESS=17.编辑系统环境变量(使用 root 用户)#vim /etc/profile追加如下内容：export JAVA_HOME=/usr/local/jdk1.7.0_05export PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$PATHexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport HADOOP_HOME=/home/hadoop/hadoop-1.0.3export PATH=$PATH:$HADOO

3、P_HOME/bin8.执行环境设置#source /etc/profile9.设置主 IP#vim hadoop-1.0.3/conf/masters把 localhost 替换成 192.168.1.24710.设置从 IP#vim hadoop-1.0.3/conf/slaves把 localhost 替换成 192.168.1.24811.设置 hdfs#vim hadoop-1.0.3/conf/hdfs-site.xml增加如下内容：dfs.name.dir/home/hadoop/hadoop-1.0.3/namedfs.data.dir/home/hadoop/hadoop-1

4、.0.3/datadfs.replication312.配置任务#vim hadoop-1.0.3/conf/mapred-site.xml增加如下内容：mapred.job.tracker192.168.1.247:9001mapred.local.dir/home/hadoop/hadoop-1.0.3/var13.配置核心文件#vim hadoop-1.0.3/conf/core-site.xmlfs.default.namehdfs:/192.168.1.247:9000fs.checkpoint.period3600fs.checkpoint.size67108864hadoop.t

5、mp.dir/home/hadoop/hadoop-1.0.3/tmp14.实现主从自动登录（单方向）创建公钥:#ssh-keygen -t rsa自己可以 ssh：#cat .ssh/id_rsa.pub .ssh/authorized_keys设置文件权限：#chmod 700 .ssh/authorized_keys测试一下，不需要密码就表示成功#ssh localhost#exit二、从服务器设置1.创建用户#useradd hadoop2.设置密码#passwd hadoop3.编辑系统环境变量(使用 root 用户)#vim /etc/profile追加如下内容：export JA

6、VA_HOME=/usr/local/jdk1.7.0_05export PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$PATHexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport HADOOP_HOME=/home/hadoop/hadoop-1.0.3export PATH=$PATH:$HADOOP_HOME/bin4.执行环境设置#source /etc/profile5.解压缩 hadoop 压缩包#tar zxvf hadoop-1.0.3.tar.gz6.设置目录权限

7、#chown -R hadoop.hadoop hadoop-1.0.3三、回到主服务器1.切换 hadoop 用户#su - hadoop2.复制公私到从服务器#ssh-copy-id -i .ssh/id_rsa.pub hadoop192.168.1.2483.测试#ssh 192.168.1.248配置正确的话是不需要密码就能登录的4.向从服务器发送配置文件#scp -r hadoop-1.0.3/conf hadoop192.168.1.248:/home/hadoop/hadoop-1.0.35.格式化分布式文件系统#hadoop-1.0.3/bin/hadoop namenode

8、 -format6.启动 hadoop 服务#hadoop-1.0.3/bin/start-dfs.sh#hadoop-1.0.3/bin/start-mapred.sh7.查看运行情况web 查询界面：http:/192.168.1.247:50030四、安装 hive(主服务器)1.压缩包装包(hadoop 用户)#tar zxvf hive-0.9.0.tar.gz#mv hive-0.9.0 hadoop-1.0.32.配置 hive 环境变量#cp hadoop-1.0.3/hive-0.9.0/conf/hive-env.sh.template hadoop-1.0.3/hive-

9、0.9.0/conf/hive-env.sh#vim hadoop-1.0.3/hive-0.9.0/conf/hive-env.sh增加一行：HADOOP_HOME=$HADOOP_HOME3.配置 hive 元数据保存到 mysql建立数据数用户，数据库使用 latin1 字符集：mysqlCREATE DATABASE hive CHARACTER SET latin1;mysqlGRANT ALL PRIVILEGES ON hive.* TO hive% IDENTIFIED BY hivepasswd;修改配置文件：#cp hadoop-1.0.3/hive-0.9.0/conf/

10、hive-default.xml.template hadoop-1.0.3/hive-0.9.0/conf/hive-site.xml#vim hadoop-1.0.3/hive-0.9.0/conf/hive-site.xml修改四个地方：javax.jdo.option.ConnectionURLjdbc:mysql:/192.168.1.247:3306/hive?createDatabaseIfNotExist=trueJDBC connect string for a JDBC metastorejavax.jdo.option.ConnectionDriverNamecom.my

11、sql.jdbc.DriverDriver class name for a JDBC metastorejavax.jdo.option.ConnectionUserNamehiveusername to use against metastore databasejavax.jdo.option.ConnectionPasswordhivepasswdpassword to use against metastore database5.复制 mysql 连接库包#cp mysql-connector-java-5.1.11-bin.jar hadoop-1.0.3/hive-0.9.0/

12、lib4.启动 hive#hadoop-1.0.3/hive-0.9.0/bin/hive5.去掉 log4j 过期警告#cp hadoop-1.0.3/hive-0.9.0/conf/hive-log4j.properties.template hadoop-1.0.3/hive-0.9.0/conf/hive-log4j.properties#vim hadoop-1.0.3/hive-0.9.0/conf/hive-log4j.properties找到“log4j.appender.EventCounter”将他的值变更为：“org.apache.hadoop.log.metrics.EventCounter”五、安装 php 访问 hive 数据库插件1.解压缩 thrift 包#tar zxvf thrift-0.8.0.tar.gz2.不要 ruby 支持否则会报错#cd thrift-0.8.0#./configure -without-ruby#make /run queries, metadata calls etc$client-execute(show tables);var_dump($client-fetchAll();$transport-close();?