Apache Ranger audit log到HDFS時出現java.net.UnknownHostException

這兩天在測試apache ranger對apache knox做audit log的時候出現了unknownHost的錯誤

2016-07-07 10:15:04,025 ERROR queue.AuditFileSpool (AuditFileSpool.java:logError(710)) - Error sending logs to consumer. provider=knox.async.batch, consumer=knox.async.batch.hdfs
2016-07-07 10:16:04,130 ERROR provider.BaseAuditHandler (BaseAuditHandler.java:logError(329)) - Error writing to log file.
java.lang.IllegalArgumentException: java.net.UnknownHostException: mycluster
        at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:411)
        at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:311)
        at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:150)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
        at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:221)
        at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:123)
        at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:890)
        at org.apache.ranger.audit.queue.AuditFileSpool.runDoAs(AuditFileSpool.java:838)
        at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:759)
        at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:757)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1689)
        at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:765)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: mycluster
        ... 22 more

因為我的namenode有做HA,看錯誤是找不到我做HA之後的FQDN name:mycluster
但是這個錯誤在對HDFS還是Hive做audit的時候並沒有發生
後來去HDP社群問了一下
https://community.hortonworks.com/questions/43669/ranger-audit-knox-log-to-hdfs.html

Ranger在對Service類的操作進行audit到HDFS動作的時後必須另外設定
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_Ranger_Install_Guide/content/save_audits_to_hdfs.html

有三個步驟要做

  1. 修改core-site.xml
  2. link core-site.xml跟hdfs-site.xml到/etc/$component/conf 下
  3. 重開該service

做了上面三個動作之後就成功解決問題了

comments powered by Disqus