最近在研究如何实现Database as a Service，给有状态的应用提供云原生持久化存储方案是最重要的因素，来自CNCF的孵化项目Rook就映入眼帘了。

Rook是什么，要解决什么问题

First thing first，Rook is not a CSI driver. —— 首先，Rook不是一个容器存储驱动。

官方对于Rook的定义是这样的：

Rook is an open source cloud-native storage orchestrator, providing the platform, framework, and support for a diverse set of storage solutions to natively integrate with cloud-native environments.

Rook turns storage software into self-managing, self-scaling, and self-healing storage services. It does this by automating deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management. Rook uses the facilities provided by the underlying cloud-native container management, scheduling and orchestration platform to perform its duties.

Rook integrates deeply into cloud native environments leveraging extension points and providing a seamless experience for scheduling, lifecycle management, resource management, security, monitoring, and user experience.

翻译过来概况下

Rook是一个开源的云原生存储编排系统，提供平台、框架和支持，提供了一套多样化的存储解决方案，可以与云原生环境进行天然集成。Rook利用云原生容器管理、调度和调度平台提供的设施，将存储软件转化为自我管理、自我扩展和自我修复的存储服务，实现自动化部署、启动、配置、扩容、升级、迁移、灾难恢复、监控和资源管理。Rook的快速扩展的特点，深度集成到云原生环境中，并在调度、生命周期管理、资源管理、安全、监控等方面提供优异的用户体验。

因此，Rook解决的问题是：

  • 快速部署一套云原生存储集群；
  • 平台化管理云原生存储集群，包括存储的扩容、升级、监控、灾难恢复等全生命周期管理；
  • 本身基于云原生容器管理（如Kubernetes），管理方便。

通过Rook部署Ceph集群

目前Rook支持多种存储集群的部署，包括：

  • Ceph，它是一个高度可扩展的分布式存储解决方案，适用于块存储、对象存储和共享文件系统，具有多年的生产部署经验。
  • EdgeFS，它是高性能和容错的分散式数据结构，可以通过对象、文件、NoSQL和块存储形式进行访问。
  • Cassandra，它是一个高度可用的NoSQL数据库，具有闪电般快速的性能、灵活可控的数据一致性和大规模的可扩展性。
  • CockroachDB，它是一个云原生的SQL数据库，用于构建全局性的、可扩展的云服务，可在灾难中生存。
  • NFS，它允许远程主机通过网络挂载文件系统，并与这些文件系统进行交互，就像在本地挂载一样。
  • YugabyteDB，是一个高性能的云端分布式SQL数据库，可以自动容忍磁盘、节点、区域和区域故障。

其中对于Ceph和EdgeFS已经是stable了，可以逐步生产使用。今天就来部署一把存储界的Super Star——Ceph。

部署前准备

官方给出了部署条件，主要是针对Kubernetes集群和节点系统层如何支持Ceph的部署条件。我这边使用的CentOS 7.6的官方系统，作了如下操作：

  1. 确保部署节点都安装了lvm2，可以通过yum install lvm2安装
  2. 如果你跟我一样，计划使用Ceph作为rbd存储，确保部署节点都安装了rbd内核模块，可以通过modprobe rbd 检查是否已安装

部署Ceph集群

所有的部署所需的物料已经都在Rook官方的Git仓库中，建议git clone最新稳定版，然后可以参照官方文档一步步进行部署。以下是我这边的部署效果。

> git clone --single-branch --branch release-1.2 https://github.com/rook/rook.git

# go to the ceph folder
> cd [ur-root/path]/rook/cluster/examples/kubernetes/ceph

# create service accounts, roles, role bindings
> kubectl create -f common.yaml

# create rook-ceph operator
> kubectl create -f operator.yaml

# create single-node ceph cluster for test
> kubectl create -f cluster-test.yaml 

# Once it is completed(it took 5 mins, which depends on ur network condition), it should look like as below:

# all the pods are deployed in `rook-ceph` namespace
> kubectl get pod -n rook-ceph
NAME                                               READY   STATUS      RESTARTS   AGE
csi-cephfsplugin-57v4k                             3/3     Running     0          6h52m
csi-cephfsplugin-g5hkg                             3/3     Running     0          6h52m
csi-cephfsplugin-provisioner-85979bcd-fpjd2        4/4     Running     0          6h52m
csi-cephfsplugin-provisioner-85979bcd-hqrsn        4/4     Running     0          6h52m
csi-cephfsplugin-x7wqp                             3/3     Running     0          6h52m
csi-rbdplugin-dmx4c                                3/3     Running     0          6h52m
csi-rbdplugin-provisioner-66f64ff49c-rr2mf         5/5     Running     0          6h52m
csi-rbdplugin-provisioner-66f64ff49c-wvpkg         5/5     Running     0          6h52m
csi-rbdplugin-ql5bc                                3/3     Running     0          6h52m
csi-rbdplugin-z8s86                                3/3     Running     0          6h52m
rook-ceph-crashcollector-node01-df5489c87-92vqj    1/1     Running     0          6h41m
rook-ceph-crashcollector-node02-679698c444-psjss   1/1     Running     0          6h47m
rook-ceph-crashcollector-node03-68c85fc9bf-59ssv   1/1     Running     0          6h43m
rook-ceph-mgr-a-d868c4664-vdhfx                    1/1     Running     0          6h48m
rook-ceph-mon-a-96cc84fc6-whb4b                    1/1     Running     0          6h48m
rook-ceph-operator-658dfb6cc4-rnnx6                1/1     Running     0          7h
rook-ceph-osd-0-668f559469-t5m7g                   1/1     Running     0          6h47m
rook-ceph-osd-1-f67d67954-48mm7                    1/1     Running     0          6h43m
rook-ceph-osd-2-7cdf548b84-x5v6g                   1/1     Running     0          6h41m
rook-ceph-osd-prepare-node01-zgbn5                 0/1     Completed   0          6h47m
rook-ceph-osd-prepare-node02-x9lts                 0/1     Completed   0          6h47m
rook-ceph-osd-prepare-node03-zsz6c                 0/1     Completed   0          6h47m
rook-discover-2k57h                                1/1     Running     0          6h52m
rook-discover-8frhx                                1/1     Running     0          6h52m
rook-discover-jdzjz                                1/1     Running     0          6h52m

部署完成后，可以通过官方提供的toolbox(就在刚才的git目录下)检查Ceph集群的健康状况：

# create ceph toolbox for check
> kubectl create -f toolbox.yaml 

# enter the pod to run ceph command
> kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') -- bash
# test 1
[root@rook-ceph-tools-55d5c49f79-jkhhs /]# ceph status
  cluster:
    id:     cda62855-74ec-4aee-9037-26445f2538ba
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum a (age 6h)
    mgr: a(active, since 6h)
    osd: 3 osds: 3 up (since 6h), 3 in (since 6h)

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   21 GiB used, 30 GiB / 51 GiB avail
    pgs:
 # test 2
[root@rook-ceph-tools-55d5c49f79-jkhhs /]# ceph osd status
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
| id |  host  |  used | avail | wr ops | wr data | rd ops | rd data |   state   |
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
| 0  | node02 | 7332M | 9.82G |    0   |     0   |    0   |     0   | exists,up |
| 1  | node03 | 7441M | 9952M |    0   |     0   |    0   |     0   | exists,up |
| 2  | node01 | 6857M | 10.2G |    0   |     0   |    0   |     0   | exists,up |
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
# test 3
[root@rook-ceph-tools-55d5c49f79-jkhhs /]# ceph df
RAW STORAGE:
    CLASS     SIZE       AVAIL      USED       RAW USED     %RAW USED
    hdd       51 GiB     30 GiB     21 GiB       21 GiB         41.45
    TOTAL     51 GiB     30 GiB     21 GiB       21 GiB         41.45

POOLS:
    POOL     ID     STORED     OBJECTS     USED     %USED     MAX AVAIL
# test 4
[root@rook-ceph-tools-55d5c49f79-jkhhs /]# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR

total_objects    0
total_used       21 GiB
total_avail      30 GiB
total_space      51 GiB

使用Ceph集群

Ceph是能提供对象存储、块存储、共享文件系统多种存储形式，这里使用块存储，兼容性更好，灵活性更高。

# go to the ceph csi rbd folder
> cd [ur-root/path]/rook/cluster/examples/kubernetes/ceph/csi/rbd

# create ceph rdb storageclass for test
> kubectl apply -f storageclass-test.yaml
cephblockpool.ceph.rook.io/replicapool created
storageclass.storage.k8s.io/rook-ceph-block created

> kubectl get storageclass
NAME              PROVISIONER                  RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-ceph-block   rook-ceph.rbd.csi.ceph.com   Delete          Immediate           true                   4m7s

有了Ceph StorageClass，我们只需要申明PVC，就可以快速按需创建出一个块设备以及对应的PV，相比传统的需要手动首先创建PV，然后在声明对应的PVC，操作更简单，管理更方便。

下面是一个基于Ceph StorageClass的PVC yaml例子：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-pv-claim
spec:
  storageClassName: rook-ceph-block # this is created sc
  accessModes:
    - ReadWriteOnce # pay attention to this value
  resources:
    requests:
      storage: 1Gi

部署PVC，并观察PV是否自动创建：

> kubectl apply -f pvc.yaml
persistentvolumeclaim/ceph-pv-claim created

> kubectl get pvc
NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
ceph-pv-claim   Bound    pvc-fb2d6d97-d7aa-43df-808c-81f15e7a2797   1Gi        RWO            rook-ceph-block   3s

> kubectl get pvc,pv
NAME                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
persistentvolumeclaim/ceph-pv-claim   Bound    pvc-fb2d6d97-d7aa-43df-808c-81f15e7a2797   1Gi        RWO            rook-ceph-block   9s

# the pv has been created automatically and bound to the pvc we created
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS      REASON   AGE
persistentvolume/pvc-fb2d6d97-d7aa-43df-808c-81f15e7a2797   1Gi        RWO            Delete           Bound    default/ceph-pv-claim   rook-ceph-block            7s

创建一个基于PVC的Pod：

apiVersion: v1
kind: Pod
metadata:
  name: ceph-pv-pod
spec:
  volumes:
    - name: ceph-pv-storage
      persistentVolumeClaim:
        claimName: ceph-pv-claim # use the pvc we created
  containers:
    - name: nginx
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html" # mount the pv
          name: ceph-pv-storage

等待Pod部署完成，观察pod的存储挂载情况：

> kubectl exec -ti ceph-pv-pod -- bash
> df -h
Filesystem               Size  Used Avail Use% Mounted on
overlay                   39G  4.1G   35G  11% /
tmpfs                     64M     0   64M   0% /dev
tmpfs                    1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mapper/centos-root   17G  7.3G  9.8G  43% /etc/hosts
/dev/mapper/centos-data   39G  4.1G   35G  11% /etc/hostname
shm                       64M     0   64M   0% /dev/shm
# the block storage is mount to the pod
/dev/rbd0                976M  2.6M  958M   1% /usr/share/nginx/html
#
tmpfs                    1.9G   12K  1.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                    1.9G     0  1.9G   0% /proc/acpi
tmpfs                    1.9G     0  1.9G   0% /proc/scsi
tmpfs                    1.9G     0  1.9G   0% /sys/firmware

> cp -R /etc/ /usr/share/nginx/html/ # make some data into block storage

通过Ceph toolbox观察Ceph集群的使用情况:

> kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') -- bash 
> ceph status
  cluster:
    id:     cda62855-74ec-4aee-9037-26445f2538ba
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum a (age 11h)
    mgr: a(active, since 11h)
    osd: 3 osds: 3 up (since 10h), 3 in (since 10h)

  data:
    pools:   1 pools, 8 pgs
    objects: 20 objects, 21 MiB
    usage:   21 GiB used, 30 GiB / 51 GiB avail
    pgs:     8 active+clean

# we have some io load now
  io:
    client:   56 KiB/s wr, 0 op/s rd, 10 op/s wr

> ceph osd status
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
| id |  host  |  used | avail | wr ops | wr data | rd ops | rd data |   state   |
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
| 0  | node02 | 7336M | 9.82G |    0   |     0   |    0   |     0   | exists,up |
| 1  | node03 | 7447M | 9946M |    0   |     0   |    0   |     0   | exists,up |
| 2  | node01 | 6871M | 10.2G |    0   |     0   |    0   |   819   | exists,up |
+----+--------+-------+-------+--------+---------+--------+---------+-----------+

理解Access Mode属性

存储系统的访问安全控制在Kubernetes的时代得到了长足的进步，远远胜于纯Docker时代的简单粗暴。来看下Kubernetes在管理存储（PV、PVC）时提供了哪些访问控制机制：

  • RWO: ReadWriteOnce，只有单个节点可以挂载这个volume，进行读写操作；
  • ROX: ReadOnlyMany，多个节点可以挂载这个volume，只能进行读操作；
  • RWX: ReadWriteMany，多个节点可以挂载这个volume，读写操作都是允许的。

所以RWO、ROX和RWX只跟同时使用volume的worker节点数量有关，而不是跟pod数量!

以前苦于没有部署云原生存储系统，一直没法实践这些特性，这次得益于Rook的便捷性，赶紧来尝鲜下。计划测试两个场景：

  • 测试ReadWriteOnce，测试步骤如下：
    1. 首先部署一个使用ReadWriteOnce访问权限的PVC的名为ceph-pv-pod的单个pod实例
    2. 然后部署一个使用相同PVC的名为n2的deployment，1个pod实例
    3. 扩容n2至6个pod副本
    4. 观察结果
> kubectl get pod -o wide --sort-by=.spec.nodeName | grep -E '^(n2|ceph)'
NAME                  READY   STATUS              IP                NODE  
n2-7db787d7f4-ww2fp   0/1     ContainerCreating   <none>            node01
n2-7db787d7f4-8r4n4   0/1     ContainerCreating   <none>            node02
n2-7db787d7f4-q5msc   0/1     ContainerCreating   <none>            node02
n2-7db787d7f4-2pfvd   1/1     Running             100.96.174.137    node03
n2-7db787d7f4-c8r8k   1/1     Running             100.96.174.139    node03
n2-7db787d7f4-hrwv4   1/1     Running             100.96.174.138    node03
ceph-pv-pod           1/1     Running             100.96.174.135    node03

从上面的结果可以看到，由于ceph-pv-pod这个Pod优先绑定了声明为ReadWriteOnce的PVC，它所在的节点node03就能成功部署n2的pod实例，而调度到其他节点的n2就无法成功部署了，挑个看看错误信息：

> kubectl describe pod n2-7db787d7f4-ww2fp
...
Events:
  Type     Reason              Age                  From                     Message
  ----     ------              ----                 ----                     -------
  Normal   Scheduled           <unknown>            default-scheduler        Successfully assigned default/n2-7db787d7f4-ww2fp to node01
  Warning  FailedAttachVolume  10m                  attachdetach-controller  Multi-Attach error for volume "pvc-fb2d6d97-d7aa-43df-808c-81f15e7a2797" Volume is already used by pod(s) n2-7db787d7f4-c8r8k, ceph-pv-pod, n2-7db787d7f4-2pfvd, n2-7db787d7f4-hrwv4

从Pod Events中可以明显看到错误了，由于ReadWriteOnce的存在，无法使用Multi-Attach了，符合期待。

  • 测试ReadWriteMany，测试步骤如下：
    1. 首先部署一个使用 ReadWriteMany访问权限的PVC的名为2ceph-pv-pod的单个pod实例
    2. 然后部署一个使用相同PVC的名为n3的deployment，1个pod实例
    3. 扩容n3至6个pod副本
    4. 观察结果

原来是想直接改第一个测试场景的创建pvc的yaml，发现如下错误。意思是创建好的pvc除了申请的存储空间以外，其他属性是无法修改的。

> k apply -f pvc.yaml 
The PersistentVolumeClaim "ceph-pv-claim" is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims

只能重新创建了。。。但当声明创建新的PVC时，又发生了问题，pvc一直处于pending状态。。。

> kubectl get pvc
NAME            STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
ceph-pv-claim   Bound     pvc-fb2d6d97-d7aa-43df-808c-81f15e7a2797   1Gi        RWO            rook-ceph-block   36h
ceph-pvc-2      Pending                                                                        rook-ceph-block   10m
> kubectl describe pvc ceph-pvc-2
...
Events:
  Type     Reason                Age                   From                                                                                                        Message
  ----     ------                ----                  ----                                                                                                        -------
  Normal   Provisioning          4m41s (x11 over 13m)  rook-ceph.rbd.csi.ceph.com_csi-rbdplugin-provisioner-66f64ff49c-wvpkg_b78217fb-8739-4ced-9e18-7430fdde964b  External provisioner is provisioning volume for claim "default/ceph-pvc-2"
  Warning  ProvisioningFailed    4m41s (x11 over 13m)  rook-ceph.rbd.csi.ceph.com_csi-rbdplugin-provisioner-66f64ff49c-wvpkg_b78217fb-8739-4ced-9e18-7430fdde964b  failed to provision volume with StorageClass "rook-ceph-block": rpc error: code = InvalidArgument desc = multi node access modes are only supported on rbd `block` type volumes
  Normal   ExternalProvisioning  3m4s (x42 over 13m)   persistentvolume-controller                                                                                 waiting for a volume to be created, either by external provisioner "rook-ceph.rbd.csi.ceph.com" or manually created by system administrator

查看event详细后，发现了这个错误信息：

failed to provision volume with StorageClass "rook-ceph-block": rpc error: code = InvalidArgument desc = multi node access modes are only supported on rbd `block` type volumes

翻译过来的意思是：多节点访问模式只支持在rbd block类型的volume上配置。。。难道说ceph的这个rbd storageclass是个假的“块存储”。。。

一般发生这种不所措的错误，首先可以去官方Github的issue或pr里找找有没有类似的问题。经过一番搜索，找到一个maintainer的相关说法。如下图所示。意思是不推荐在ceph rbd模式下使用RWX访问控制，如果应用层没有访问锁机制，可能会造成数据损坏。

进而找到了官方上的说法

There are two CSI drivers integrated with Rook that will enable different scenarios:

  • RBD: This driver is optimized for RWO pod access where only one pod may access the storage
  • CephFS: This driver allows for RWX with one or more pods accessing the same storage

好吧，原来官方网站已经说明了CephFS模式是使用RWX模式的正确选择。

使用CephFS测试ReadWriteMany(RWX)模式

官方已经提供了支持CephFS的StorageClass，我们需要部署开启：

> cd [ur-rook-git]/rook/cluster/examples/kubernetes/ceph/csi/cephfs/
> kubectl apply -f storageclass.yaml
> kubectl get sc
NAME              PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION
csi-cephfs        rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                
rook-ceph-block   rook-ceph.rbd.csi.ceph.com      Delete          Immediate           true

注意观察上面的StorageClass文件parameters里有一行：

# CephFS filesystem name into which the volume shall be created
  fsName: myfs

这是使用CephFS必须要有的文件系统，必须创建好，才能声明创建基于CephFS的PV和PVC。官方提供了示例：

> cat cephfs-filesystem.yaml
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: myfs # the name should be the same as the one in storageclass yaml
  namespace: rook-ceph
spec:
  metadataPool:
    replicated:
      size: 3
  dataPools:
    - replicated:
        size: 3
  preservePoolsOnDelete: true
  metadataServer:
    activeCount: 1
    activeStandby: true

> kubectl apply -f cephfs-filesystem.yaml

创建完CephFS的StorageClass和FileSystem，就可以测试了。测试场景为部署一个deployment，6个副本，使用RWX模式的Volume：

# pvc-cephfs.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cephfs-pvc
spec:
  storageClassName: csi-cephfs # new storageclass
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
---
# deployment-cephfs-pvc.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: n4-cephfs
    pv: cephfs
  name: n4-cephfs
spec:
  replicas: 3
  selector:
    matchLabels:
      app: n4-cephfs
      pv: cephfs
  template:
    metadata:
      labels:
        app: n4-cephfs
        pv: cephfs
    spec:
      volumes:
        - name: fsceph-pv-storage
          persistentVolumeClaim:
            claimName: cephfs-pvc
      containers:
      - image: nginx
        name: nginx
        ports:
          - containerPort: 80
            name: "http-server"
        volumeMounts:
          - mountPath: "/usr/share/nginx/html"
            name: fsceph-pv-storage

部署后观察每个pod的运行情况以及PV和PVC创建情况：

> kubectl get pod -l pv=cephfs -o wide
NAME                         READY   STATUS     IP                NODE  
n4-cephfs-859b956b65-768z7   1/1     Running    100.95.185.225    node02
n4-cephfs-859b956b65-cs9jn   1/1     Running    100.96.174.143    node03
n4-cephfs-859b956b65-rwdn4   1/1     Running    100.117.144.146   node01
n4-cephfs-859b956b65-x55s4   1/1     Running    100.96.174.142    node03
n4-cephfs-859b956b65-xffwb   1/1     Running    100.117.144.148   node01
n4-cephfs-859b956b65-xx8nz   1/1     Running    100.117.144.147   node01

> 
> kubectl get pvc,pv
NAME                                  STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
persistentvolumeclaim/cephfs-pvc      Bound     pvc-75b40dd7-b880-4d67-9da6-88aba8616466   1Gi        RWX            csi-cephfs

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS
persistentvolume/pvc-75b40dd7-b880-4d67-9da6-88aba8616466   1Gi        RWX            Delete           Bound    default/cephfs-pvc      csi-cephfs

分布在不同的节点上的pod都能部署成功，PV也能创建绑定成功。符合测试预期。

更深入地观察存储挂载机制

通过上面两个测试场景，我们来看下背后的云原生存储的运行逻辑：

  • 进入pod观察存储挂载情况

对比两个测试场景pod实例里面存储挂载情况：

# use ceph as rbd storage
# it is mount as block device 
> df -h
/dev/rbd0                976M  3.3M  957M   1% /usr/share/nginx/html

# use ceph as file system storage
# it is mount as nfs storage
> df -h 
10.109.80.220:6789:/volumes/csi/csi-vol-1dc92634-79cd-11ea-96a3-26ab72958ea2  1.0G     0  1.0G   0% /usr/share/nginx/html

可以看到Ceph rbd和CephFS挂载到Pod里的方式是有差别的。

  • 观察主机层存储挂载情况
# use ceph as rbd storage
> df -h |grep rbd # on work node
/dev/rbd0                                                                     976M  3.3M  957M   1% /var/lib/kubelet/pods/e432e18d-b18f-4b26-8128-0b0219a60662/volumes/kubernetes.io~csi/pvc-fb2d6d97-d7aa-43df-808c-81f15e7a2797/mount

# use ceph as file system storage
> df -h |grep csi
10.109.80.220:6789:/volumes/csi/csi-vol-1dc92634-79cd-11ea-96a3-26ab72958ea2  1.0G     0  1.0G   0% /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-75b40dd7-b880-4d67-9da6-88aba8616466/globalmount

简单解释下主机层相关路径的命名规则：

/var/lib/kubelet/pods/<Pod的ID>/volumes/kubernetes.io~<Volume类型>/<Volume名字>

最终都是通过docker run映射到容器里去：

docker run -v /var/lib/kubelet/pods/<Pod-ID>/volumes/kubernetes.io~<Volume类型>/<Volume名字>:/<容器内目标目录> 镜像 ...
  • 从Kubernetes观察存储挂载情况

Kubernetes提供了获取StorageClass、PV和Node之间的关系——volumeattachment资源类型。它的官方解释是：

VolumeAttachment captures the intent to attach or detach the specified volume to/from the specified node. VolumeAttachment objects are non-namespaced.

来看下当前的情况：

> kubectl get volumeattachment
NAME                                                                   ATTACHER                        PV                                         NODE     ATTACHED   AGE
csi-0453435f600da9580d6dff3bceb3d151d36462fa001c682663dd7371beff309f   rook-ceph.cephfs.csi.ceph.com   pvc-75b40dd7-b880-4d67-9da6-88aba8616466   node02   true       13h
csi-89e8c56d49e524d5d8dd6bfe4809f040f35f8b416a09241bcb9e6632ced232be   rook-ceph.cephfs.csi.ceph.com   pvc-75b40dd7-b880-4d67-9da6-88aba8616466   node03   true       13h
csi-97bab907614b96ddf2df6ce15d547e409b68bb41591f1b4bd25eacb9a59ec9de   rook-ceph.rbd.csi.ceph.com      pvc-fb2d6d97-d7aa-43df-808c-81f15e7a2797   node03   true       2d3h
csi-f002b45a7930af0b144fd0d40734ec3a41a207b892db471cb82d373b9bfd79bd   rook-ceph.cephfs.csi.ceph.com   pvc-75b40dd7-b880-4d67-9da6-88aba8616466   node01   true       13h

能看到每个主机层挂载点的详细情况，方便大家troubleshooting。

Ceph界面化管理Ceph Dashboard

Rook官方很贴心地提供了Ceph界面化管理的解决方案——Ceph dashboard。标准版部署Rook已经自带这个功能，默认是无法集群外访问的，手动expose为nodeport模式即可：

> kubectl -n rook-ceph get svc |grep dash
rook-ceph-mgr-dashboard                  ClusterIP   10.97.126.41     <none>        8443/TCP            2d14h
rook-ceph-mgr-dashboard-external-https   NodePort    10.109.153.31    <none>        8443:30010/TCP      14h

通过浏览器访问https://node-ip:30010，默认登录用户名为admin，密码可以通过这样的方式获取：

kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo

登录后，界面如下。内容非常多，包括读写速率监控，健康监控等，绝对是Ceph管理的好帮手。

还提供交互式API文档，非常贴心。

戳视频可以看完整Demo：

总结

Rook能帮你快速搭建一套Production-Ready的云原生存储平台，同时提供全生命周期管理，适合初中高级全阶段的存储管理玩家。

本文涉及的部署物料可以去这里获取：

https://github.com/nevermosby/rook-ceph-tutorial

微海报分享

1 对 “Rook带你玩转云原生存储”的想法；

发表评论

邮箱地址不会被公开。 必填项已用*标注