Kubernetes创建基于Ceph rbd的持久存储问题

  1. ceph集群已经正常部署,基于admin认证都可以通过
  2. 创建需要的 foo image
  3. 在k8s里正确注入admin的secret


以下pod的yaml文件
apiVersion: "v1"
kind: "Pod"
metadata: 
name: "rbd2"
spec: 
containers: 

  name: "rbd-rw"
  image: "nginx"
  resources:
    requests:
      memory: "64Mi"
      cpu: "250m"
    limits:
      memory: "128Mi"
      cpu: "500m"
  volumeMounts: 
    - 
      mountPath: "/mnt/rbd"
      name: "rbdpd"
volumes: 

  name: "rbdpd"
  rbd: 
    monitors: 
      - "10.10.0.109:6789"
      - "10.10.0.110:6789"
      - "10.10.0.111:6789"
    pool: "rbd"
    image: "foo"
    user: "admin"
    secretRef: 
      name: "ceph-client-admin-keyring"
    fsType: "ext4"
    keyring: "/etc/ceph/ceph.client.admin.keyring"
    readOnly: true

  • 如果按以上的yaml文件创建pod,默认情况创建不成功
    Dec 21 13:22:04 docker kubelet: E1221 13:22:04.105584 13909 pod_workers.go:112] Error syncing pod 200bcc0d-a797-11e5-98a5-fa163e29fbd9, skipping: exit status 1
    Dec 21 13:22:13 docker kernel: libceph: client8060 fsid 655807bb-c157-46e6-8f10-2454da8d710f
    Dec 21 13:22:13 docker kernel: libceph: auth method 'x' error -1
    Dec 21 13:22:13 docker kernel: libceph: client9042 fsid 655807bb-c157-46e6-8f10-2454da8d710f
    Dec 21 13:22:13 docker kernel: libceph: auth method 'x' error -1
    Dec 21 13:22:13 docker kernel: libceph: client8032 fsid 655807bb-c157-46e6-8f10-2454da8d710f
    Dec 21 13:22:13 docker kernel: libceph: auth method 'x' error -1
    Dec 21 13:22:13 docker kubelet: E1221 13:22:13.992751 13909 disk_manager.go:55] failed to attach disk
    Dec 21 13:22:13 docker kubelet: E1221 13:22:13.992792 13909 rbd.go:203] rbd: failed to setup
    Dec 21 13:22:13 docker kubelet: E1221 13:22:13.992862 13909 kubelet.go:1360] Unable to mount volumes for pod "rbd2_default": exit status 1; skipping pod
    Dec 21 13:22:14 docker kubelet: E1221 13:22:14.011721 13909 pod_workers.go:112] Error syncing pod 200bcc0d-a797-11e5-98a5-fa163e29fbd9, skipping: exit status 1
  • 如果我预先在目标宿主机上把foo挂载(通过rbd map),即可创建成功 pod,但rbd 是只读的
    root@rbd2:/# touch /mnt/rbd/test.txt
    touch: cannot touch '/mnt/rbd/test.txt': Read-only file system
    root@rbd2:/#
  • 如果我把yaml里的字段 readOnly: true 改为 false,pod创建失败。

  • ### 这是k8s v1.1的bug ?还是我姿势不对? 另外,我通过pvpvc的方式也是创建失败?跪等高人指点。
已邀请:

要回复问题请先登录注册