第117课： Spark Streaming性能优化：如何最大程度的确保Spark Cluster和Kafka链接的稳

第117课：  Spark Streaming性能优化：如何最大程度的确保Spark Cluster和Kafka链接的稳定性

1 Spark Streaming与Kafka连接问题
2 KafkaReceiver

spark2.x弄了一个静态类型检查，python就不能用

ZkUtils.scala
zookeeper.session.timeout.ms 系统默认6000 6s，这个超时zk就资源重新分配，所有数据都不能接收。考虑GC等因素，zookeeper.seesion.timeout.ms这个值在生产环境中设置为30s。30000  ACK确认

 

 

ZkUtils

class ZKConfig(props: VerifiableProperties) { 
/** ZK host string */ 
val zkConnect = props.getString("zookeeper.connect")
 
/** zookeeper session timeout */ 
val zkSessionTimeoutMs = props.getInt("zookeeper.session.timeout.ms", 6000)  

/** the max time that the client waits to establish a connection to zookeeper */ 
val zkConnectionTimeoutMs = props.getInt("zookeeper.connection.timeout.ms",zkSessionTimeoutMs)
 
/** how far a ZK follower can be behind a ZK leader */ 
val zkSyncTimeMs = props.getInt("zookeeper.sync.time.ms", 2000)}

 

 

ZookeeperConsumerConnector.scala