Hadoop配置机架感知(Python脚本)

来源:http://www.boundlesssupremacy.com 作者:win7 人气:98 发布时间:2019-08-07
摘要:2012-06-08 14:50:42,728 INFO org.apache.hadoop.net.NetworkTopology:Adding a new node: /rack2/hadoop-node-52 2012-06-08 14:50:42,731 INFO org.apache.hadoop.mapred.JobTracker: Addingtracker tracker_hadoop-node-52:localhost.localdomain/127.0.0

2012-06-08 14:50:42,728 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/hadoop-node-52
2012-06-08 14:50:42,731 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker tracker_hadoop-node-52:localhost.localdomain/127.0.0.1:33999 to host hadoop-node-52
2012-06-08 14:50:42,753 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack1/hadoop-node-34
2012-06-08 14:50:42,754 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker tracker_hadoop-node-34:localhost.localdomain/127.0.0.1:37025 to host hadoop-node-34
2012-06-08 14:50:42,773 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/hadoop-node-51

    2)编写 RackAware.py 脚本:

if __name__=="__main__":
        print "/" rack.get(sys.argv[1],"rack0")

hadoop集群互联网品质优化:hadoop机架感知达成及铺排

更多Hadoop相关信息见Hadoop 专题页面 http://www.linuxidc.com/topicnews.aspx?tid=13

2020欧洲杯官方投注-2020欧洲杯官方投注网址 1

2012-06-08 14:42:49,492 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 2 racks and 10 datanodes
2012-06-08 14:42:49,492 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks
2020欧洲杯官方投注-2020欧洲杯官方投注网址,2012-06-08 14:42:49,642 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: First cycle completed 0 blocks in 0 msec
2012-06-08 14:42:49,642 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: Queue flush completed 0 blocks in 0 msec processing time, 0 msec clock time, 1 cycles

Hadoop集群的机架感知作用是升高hadoop集群互联网质量的基本点参数,配置hadoop机架感知作用的步调如下:

python代码摘自树蛇的博客

      1)修改core-site.xml文件,增加以下配置项:

2020欧洲杯官方投注-2020欧洲杯官方投注网址 2

       更为详细的原理及测量检验请参见:

放入一下代码

            就算布置成功,namenode运营日志中会输出:

#!/usr/bin/python
#-*-coding:UTF-8 -*-
import sys

       4)重启namenode

下一场展开conf/core-site.html

       3)试行命令:chmod x RackAware.py

前天QQ群里提了一个Hadoop运转功用分配的难点,计算一下,写个篇章。集群使用hadoop-1.0.3

  1. 2011-12-21 14:28:44,495 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack1/192.168.1.15:50010  

2012-06-08 14:42:19,174 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.49:50010 storage DS-1155827498-192.168.1.49-50010-1338289368956
2012-06-08 14:42:19,204 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/192.168.1.49:50010
2012-06-08 14:42:19,205 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.53:50010 storage DS-1773813988-192.168.1.53-50010-1338289405131
2012-06-08 14:42:19,226 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack2/192.168.1.53:50010
2012-06-08 14:42:19,226 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.34:50010 storage DS-2024494948-127.0.0.1-50010-1338289438983
2012-06-08 14:42:19,242 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack1/192.168.1.34:50010
2012-06-08 14:42:19,242 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.1.54:50010 storage DS-767528606-192.168.1.54-50010-1338289412267

       由于尚未找到适合的文书档案说明到底是主机名照旧ip地址会被流传到剧本,所以在本子中最佳包容主机名和ip地址,如若机房架构比较复杂的话,脚本能够重返如:/dc1/rack1 类似的字符串。

本文由2020欧洲杯官方投注-2020欧洲杯官方投注网址发布于win7,转载请注明出处:Hadoop配置机架感知(Python脚本)

关键词: 欧洲杯外围

上一篇:Linux Apache、汤姆cat整合实用版

下一篇:没有了

最火资讯