Zookeeper Configuration¶

Reference for zoo.cfg / zookeeper.properties parameters. See also the Apache ZooKeeper Admin Guide.

tickTime¶

tickTime is the base time unit (milliseconds) for all time-related operations inside the ensemble. It is the foundation of ZooKeeper’s timing model.

Heartbeat checks, session validation, and other scheduled tasks run in multiples of tickTime.

Treat tickTime as the cluster “heartbeat pulse”—it controls sensitivity to network latency and overall stability. Tune it for your network.

tickTime Default Value¶

# Minimum time unit / heartbeat interval (milliseconds)
tickTime=2000

tickTime Applicable Scene¶

Global (all nodes must match).

tickTime Configuration Policy¶

tickTime ≤ minimum expected session timeout / 2
tickTime ≥ maximum expected session timeout / 20

# Suggested values
# LAN: 2000–3000 ms (2–3 s)
# Cross-region / high-latency: 4000–5000 ms (4–5 s)

initLimit¶

initLimit caps how long a new or restarting follower may take to sync with the leader, expressed in tickTime units.

Think of it as the “join gate”—too low blocks joins; too high hides slow sync. Tune using zkServer.sh status and join/recovery metrics.

initLimit Default Value¶

initLimit=10

# Actual timeout = initLimit × tickTime
# Default: 10 × 2000 = 20 seconds

initLimit Applicable Scene¶

• New follower joining the ensemble
• Crashed follower reconnecting
• Post-election leader–follower initialization

initLimit Configuration Policy¶

# Constraints
initLimit > syncLimit
initLimit ≥ 2 × ping_time_interval

initLimit Possible Fault¶

syncLimit¶

syncLimit limits how long the leader waits for a follower to respond to a request (heartbeat, log push, write ack), in tickTime units.

syncLimit Default Value¶

syncLimit=5

# Actual timeout = syncLimit × tickTime
# Default: 5 × 2000 = 10 seconds

syncLimit Applicable Scene¶

• Leader → follower heartbeats
• Transaction log (zxid) replication
• Write quorum acknowledgements

syncLimit Configuration Policy¶

# Constraints
syncLimit < initLimit   # often syncLimit <= initLimit/2
# Rule of thumb
syncLimit * tickTime > average_RTT × 3

syncLimit Possible Fault¶

dataDir¶

dataDir stores in-memory database snapshots and ensemble metadata. Required for persistence.

It affects durability, recovery, and I/O. Manage it together with autopurge.* and metrics.

dataDir Default Value¶

# Do not use /tmp in production; shown for illustration only.
dataDir=/tmp/zookeeper

dataDir Contents¶

When dataLogDir is set, transaction logs go there; snapshots remain under dataDir.

dataDir Layout¶

/var/lib/zookeeper/data/
├── version-2/
│   ├── log.100000001
│   ├── snapshot.100000000
├── myid
└── acceptedEpoch

dataDir Configuration Policy¶

autopurge.snapRetainCount=3
autopurge.purgeInterval=24
dataLogDir=/disks/ssd1/zookeeper_logs

dataDir Troubleshooting¶

dataDir Recovery Tools¶

java -cp zookeeper.jar:lib/* org.apache.zookeeper.server.SnapshotFormatter snapshot.100000000
java -cp zookeeper.jar:lib/* org.apache.zookeeper.server.LogFormatter log.100000001

dataDir Backup Example¶

zk_backup_dir="/backup/$(date +%Y%m%d)"
mkdir -p $zk_backup_dir
cp $dataDir/version-2/snapshot.* $zk_backup_dir
cp $dataDir/version-2/log.* $zk_backup_dir

Related Optimizations¶

1. Separate transaction logs

   dataDir=/hdd1/zookeeper/snapshots
   dataLogDir=/ssd1/zookeeper/logs
   

2. Filesystem

   mkfs.xfs /dev/sdb1
   mount -o noatime /dev/sdb1 /var/lib/zookeeper
   

3. JVM

   JVMFLAGS="-XX:MaxDirectMemorySize=2G"
   

dataLogDir¶

Optional directory dedicated to transaction logs. Pair with dataDir for high write throughput.

dataLogDir Default Value¶

None (logs use dataDir when unset).

dataLogDir vs dataDir¶

dataLogDir Performance¶

• Why separate: avoids log sequential writes competing with snapshot random I/O.

• Hardware: put logs on fast SSD; snapshots can use HDD.

  dataLogDir=/ssd_pool/zookeeper/logs
  dataDir=/hdd_pool/zookeeper/data
  

• Recovery: log corruption is severe; snapshot corruption can often be rebuilt from logs.

dataLogDir Layout¶

/data_logs/
└── version-2/
    ├── log.100000001
    ├── log.100000002
    └── log.100000003

• Naming: log.<zxid>
• Default segment size: 64 MB (preAllocSize)

dataLogDir Configuration Policy¶

dataLogDir Maintenance¶

java -cp zookeeper.jar:lib/* \
  org.apache.zookeeper.server.LogFormatter log.100000001 | tail -100

kill -SIGUSR1 <zk_pid>

dataLogDir Faults¶

dataLogDir Recovery Flow¶

graph TB
    A[Detect log corruption] --> B[Stop ensemble]
    B --> C[Backup existing files]
    C --> D[SnapshotFormatter analysis]
    D --> E[Find last valid zxid]
    E --> F[Remove invalid segments]
    F --> G[Restart ensemble]

dataLogDir Summary¶

• Core to write performance
• Physically separate from dataDir in production
• Use SSD for logs
• Alert above ~80% disk usage
• Periodically validate logs with LogFormatter

clientPort¶

Port for client connections.

clientPort Default Value¶

clientPort=2181

maxClientCnxns¶

Maximum concurrent client connections per IP (default often 60). Increase if you expect more clients from one host.

maxClientCnxns Default Value¶

# maxClientCnxns=60

autopurge.snapRetainCount¶

Number of snapshots to retain under dataDir. See maintenance.

autopurge.snapRetainCount Default Value¶

# autopurge.snapRetainCount=3

autopurge.purgeInterval¶

Purge interval in hours. Set 0 to disable automatic purge.

autopurge.purgeInterval Default Value¶

# autopurge.purgeInterval=1

metricsProvider¶

Prometheus metrics exporter. See prometheus.io.

metricsProvider Default Value¶

#metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
#metricsProvider.httpHost=0.0.0.0
#metricsProvider.httpPort=7000
#metricsProvider.exportJvmInfo=true

Zookeeper Time Parameters¶

Time Parameter Relationships¶

graph LR
    A[tickTime] --> B[Heartbeat interval]
    A --> C[initLimit timeout]
    A --> D[syncLimit timeout]
    A --> E[Session timeout]

    B -->|every tickTime ms| F[Inter-server heartbeat]
    C -->|initLimit*tickTime| G[New node join window]
    D -->|syncLimit*tickTime| H[Leader-follower sync window]
    E -->|2~20*tickTime| I[Client session lifetime]

Timeout Calculations¶

Time-Related Errors¶

• Frequent session expiry: tickTime too low + jitter → increase tickTime and review session bounds.

• Join failures: ensure initLimit × tickTime exceeds actual sync time.

  ERROR [QuorumPeer[myid=3]/...] - Failed to connect to leader
  Reason: INIT_SYNC_TIMEOUT after 20000ms (tickTime=2000, initLimit=10)
  

• Sync delay warnings: ensure syncLimit × tickTime covers cross-DC RTT.

  WARN [SyncThread:3] - Unexpected packet length
  

Zookeeper Cluster Communication¶

Node Joining the Ensemble¶

sequenceDiagram
    participant NewFollower
    participant Leader

    Note over NewFollower: 1. Connect request
    NewFollower->>Leader: CONNECT_REQUEST
    Note over Leader: 2. Connect ack
    Leader-->>NewFollower: CONNECT_ACK

    Note over NewFollower: 3. Initial sync request
    NewFollower->>Leader: SYNC_REQUEST
    Note over Leader: 4. Snapshot stream
    Leader-->>NewFollower: SNAPSHOT_STREAM

    Note over NewFollower: 5. Validation
    NewFollower->>Leader: VALIDATION_REPORT
    Note over Leader: 6. Sync complete
    Leader-->>NewFollower: SYNC_COMPLETE

    Note right of Leader: Deadline: initLimit*tickTime
    alt Within deadline
        Leader-->>NewFollower: WELCOME_TO_CLUSTER
    else Timeout
        Leader-->>NewFollower: CONNECTION_DROP
    end

Cluster Heartbeat¶

sequenceDiagram
    participant Leader
    participant Follower

    Note over Leader: 1. Heartbeat / request
    Leader->>Follower: REQUEST(id=123)

    Note over Leader: Timer: syncLimit*tickTime
    rect rgb(255,240,240)
    Note over Follower: Processing...
    Follower-->>Leader: ACK(id=123)
    end

    alt ACK within syncLimit*tickTime
        Leader-->Leader: Mark follower alive
    else Timeout
        Leader-->Leader: Mark follower lost
        Leader-->Leader: Possible reconfig
    end

Zookeeper Data Write Path¶

Write Path Overview¶

graph TD
    A[Client write] --> B[Leader]
    B --> C[Write txn log]
    C --> D[Update in-memory DB]
    D --> E[Broadcast to followers]
    E --> F[Followers write local log]
    F --> G[Periodic snapshot]
    G --> H[Purge old logs]

Transaction Log Path¶

graph LR
    A[Write request] --> B[Leader]
    B --> C[Txn log dataLogDir]
    C --> D[Update memory state]
    D --> E[Snapshot dataDir]
    E --> F[Reply to client]