Automating nodetool repair for cassandra on kubernetes

by rvabdn   Last Updated May 16, 2019 02:00 AM

We are running cassandra on kubernetes using the yaml below. It's working fine except for nodetool. We would like to have a kubernetes cronjob that executes nodetool repair periodically but accessing the jmx port (7199) from outside of the cassandra pods is not working. I'm fairly sure the problem is that the jmx port is listening on 127.0.0.1 and I need it to listen on the pod ip or even better would be the node ip.

Has anyone setup something similar?

Inside the pod

[email protected]:/# nc -vz localhost 9042
localhost [127.0.0.1] 9042 (?) open
[email protected]:/# nc -vz localhost 7199
localhost [127.0.0.1] 7199 (?) open

Outside the pod

$ nc -vz localhost 9042
nc: connect to localhost port 9042 (tcp) failed: Connection refused
Connection to localhost 9042 port [tcp/*] succeeded!
$ nc -vz localhost 7199
nc: connect to localhost port 7199 (tcp) failed: Connection refused
nc: connect to localhost port 7199 (tcp) failed: Connection refused

Outside the pod using the pod ip

$ nc -vz 10.244.1.20 7199
nc: connect to 10.244.1.20 port 7199 (tcp) failed: Connection refused
$ nc -vz 10.244.1.20 9042
Connection to 10.244.1.20 9042 port [tcp/*] succeeded!

apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandra
  name: cassandra
spec:
  clusterIP: None
  ports:
  - port: 9042
  selector:
    app: cassandra
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: cassandra
  labels:
    app: cassandra
spec:
  serviceName: cassandra
  replicas: {{ .Values.global.replicas }}
  selector:
    matchLabels:
      app: cassandra
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      terminationGracePeriodSeconds: 1800
      containers:
      - name: cassandra
        image: cassandra:3.11.3
        imagePullPolicy: Always
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
          hostPort: 7199
        - containerPort: 9042
          hostPort: 9042
          name: cql
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
            cpu: "500m"
            memory: 1Gi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        lifecycle:
          preStop:
            exec:
              command: 
              - /bin/sh
              - -c
              - nodetool drain
        env:
          - name: MAX_HEAP_SIZE
            value: 512M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            value: "cassandra-0.cassandra.default.svc.cluster.local"
          - name: CASSANDRA_CLUSTER_NAME
            value: "K8Demo"
          - name: CASSANDRA_DC
            value: "DC1-K8Demo"
          - name: CASSANDRA_RACK
            value: "Rack1-K8Demo"
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        volumeMounts:
        - name: cassandra-data
          mountPath: /var/lib/cassandra/data
      volumes:
      - name: cassandra-data
        hostPath:
          path: /data/cassandra


Related Questions


Updated July 25, 2018 17:00 PM

Updated August 06, 2018 13:00 PM

Updated September 12, 2018 09:00 AM

Updated March 30, 2019 21:00 PM

Updated April 02, 2015 22:00 PM