最近在研究如何实现Database as a Service,给有状态的应用提供云原生持久化存储方案是最重要的因素,来自CNCF的孵化项目Rook就映入眼帘了。

Rook是什么,要解决什么问题

First thing first,Rook is not a CSI driver. —— 首先,Rook不是一个容器存储驱动。

官方对于Rook的定义是这样的:

Rook is an open source cloud-native storage orchestrator, providing the platform, framework, and support for a diverse set of storage solutions to natively integrate with cloud-native environments.

Rook turns storage software into self-managing, self-scaling, and self-healing storage services. It does this by automating deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management. Rook uses the facilities provided by the underlying cloud-native container management, scheduling and orchestration platform to perform its duties.

Rook integrates deeply into cloud native environments leveraging extension points and providing a seamless experience for scheduling, lifecycle management, resource management, security, monitoring, and user experience.

翻译过来概况下

Rook是一个开源的云原生存储编排系统,提供平台、框架和支持,提供了一套多样化的存储解决方案,可以与云原生环境进行天然集成。Rook利用云原生容器管理、调度和调度平台提供的设施,将存储软件转化为自我管理、自我扩展和自我修复的存储服务,实现自动化部署、启动、配置、扩容、升级、迁移、灾难恢复、监控和资源管理。Rook的快速扩展的特点,深度集成到云原生环境中,并在调度、生命周期管理、资源管理、安全、监控等方面提供优异的用户体验。

因此,Rook解决的问题是:

  • 快速部署一套云原生存储集群;
  • 平台化管理云原生存储集群,包括存储的扩容、升级、监控、灾难恢复等全生命周期管理;
  • 本身基于云原生容器管理(如Kubernetes),管理方便。

通过Rook部署Ceph集群

目前Rook支持多种存储集群的部署,包括:

  • Ceph,它是一个高度可扩展的分布式存储解决方案,适用于块存储、对象存储和共享文件系统,具有多年的生产部署经验。
  • EdgeFS,它是高性能和容错的分散式数据结构,可以通过对象、文件、NoSQL和块存储形式进行访问。
  • Cassandra,它是一个高度可用的NoSQL数据库,具有闪电般快速的性能、灵活可控的数据一致性和大规模的可扩展性。
  • CockroachDB,它是一个云原生的SQL数据库,用于构建全局性的、可扩展的云服务,可在灾难中生存。
  • NFS,它允许远程主机通过网络挂载文件系统,并与这些文件系统进行交互,就像在本地挂载一样。
  • YugabyteDB,是一个高性能的云端分布式SQL数据库,可以自动容忍磁盘、节点、区域和区域故障。

其中对于Ceph和EdgeFS已经是stable了,可以逐步生产使用。今天就来部署一把存储界的Super Star——Ceph。

部署前准备

官方给出了部署条件,主要是针对Kubernetes集群和节点系统层如何支持Ceph的部署条件。我这边使用的CentOS 7.6的官方系统,作了如下操作:

  1. 确保部署节点都安装了lvm2,可以通过yum install lvm2安装
  2. 如果你跟我一样,计划使用Ceph作为rbd存储,确保部署节点都安装了rbd内核模块,可以通过modprobe rbd 检查是否已安装

部署Ceph集群

所有的部署所需的物料已经都在Rook官方的Git仓库中,建议git clone最新稳定版,然后可以参照官方文档一步步进行部署。以下是我这边的部署效果。

> git clone --single-branch --branch release-1.2 https://github.com/rook/rook.git

# go to the ceph folder
> cd [ur-root/path]/rook/cluster/examples/kubernetes/ceph

# create service accounts, roles, role bindings
> kubectl create -f common.yaml

# create rook-ceph operator
> kubectl create -f operator.yaml

# create single-node ceph cluster for test
> kubectl create -f cluster-test.yaml 

# Once it is completed(it took 5 mins, which depends on ur network condition), it should look like as below:

# all the pods are deployed in `rook-ceph` namespace
> kubectl get pod -n rook-ceph
NAME                                               READY   STATUS      RESTARTS   AGE
csi-cephfsplugin-57v4k                             3/3     Running     0          6h52m
csi-cephfsplugin-g5hkg                             3/3     Running     0          6h52m
csi-cephfsplugin-provisioner-85979bcd-fpjd2        4/4     Running     0          6h52m
csi-cephfsplugin-provisioner-85979bcd-hqrsn        4/4     Running     0          6h52m
csi-cephfsplugin-x7wqp                             3/3     Running     0          6h52m
csi-rbdplugin-dmx4c                                3/3     Running     0          6h52m
csi-rbdplugin-provisioner-66f64ff49c-rr2mf         5/5     Running     0          6h52m
csi-rbdplugin-provisioner-66f64ff49c-wvpkg         5/5     Running     0          6h52m
csi-rbdplugin-ql5bc                                3/3     Running     0          6h52m
csi-rbdplugin-z8s86                                3/3     Running     0          6h52m
rook-ceph-crashcollector-node01-df5489c87-92vqj    1/1     Running     0          6h41m
rook-ceph-crashcollector-node02-679698c444-psjss   1/1     Running     0          6h47m
rook-ceph-crashcollector-node03-68c85fc9bf-59ssv   1/1     Running     0          6h43m
rook-ceph-mgr-a-d868c4664-vdhfx                    1/1     Running     0          6h48m
rook-ceph-mon-a-96cc84fc6-whb4b                    1/1     Running     0          6h48m
rook-ceph-operator-658dfb6cc4-rnnx6                1/1     Running     0          7h
rook-ceph-osd-0-668f559469-t5m7g                   1/1     Running     0          6h47m
rook-ceph-osd-1-f67d67954-48mm7                    1/1     Running     0          6h43m
rook-ceph-osd-2-7cdf548b84-x5v6g                   1/1     Running     0          6h41m
rook-ceph-osd-prepare-node01-zgbn5                 0/1     Completed   0          6h47m
rook-ceph-osd-prepare-node02-x9lts                 0/1     Completed   0          6h47m
rook-ceph-osd-prepare-node03-zsz6c                 0/1     Completed   0          6h47m
rook-discover-2k57h                                1/1     Running     0          6h52m
rook-discover-8frhx                                1/1     Running     0          6h52m
rook-discover-jdzjz                                1/1     Running     0          6h52m


部署完成后,可以通过官方提供的toolbox(就在刚才的git目录下)检查Ceph集群的健康状况:

# create ceph toolbox for check
> kubectl create -f toolbox.yaml 

# enter the pod to run ceph command
> kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') -- bash
# test 1
[root@rook-ceph-tools-55d5c49f79-jkhhs /]# ceph status
  cluster:
    id:     cda62855-74ec-4aee-9037-26445f2538ba
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum a (age 6h)
    mgr: a(active, since 6h)
    osd: 3 osds: 3 up (since 6h), 3 in (since 6h)

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   21 GiB used, 30 GiB / 51 GiB avail
    pgs:
 # test 2
[root@rook-ceph-tools-55d5c49f79-jkhhs /]# ceph osd status
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
| id |  host  |  used | avail | wr ops | wr data | rd ops | rd data |   state   |
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
| 0  | node02 | 7332M | 9.82G |    0   |     0   |    0   |     0   | exists,up |
| 1  | node03 | 7441M | 9952M |    0   |     0   |    0   |     0   | exists,up |
| 2  | node01 | 6857M | 10.2G |    0   |     0   |    0   |     0   | exists,up |
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
# test 3
[root@rook-ceph-tools-55d5c49f79-jkhhs /]# ceph df
RAW STORAGE:
    CLASS     SIZE       AVAIL      USED       RAW USED     %RAW USED
    hdd       51 GiB     30 GiB     21 GiB       21 GiB         41.45
    TOTAL     51 GiB     30 GiB     21 GiB       21 GiB         41.45

POOLS:
    POOL     ID     STORED     OBJECTS     USED     %USED     MAX AVAIL
# test 4
[root@rook-ceph-tools-55d5c49f79-jkhhs /]# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR

total_objects    0
total_used       21 GiB
total_avail      30 GiB
total_space      51 GiB

使用Ceph集群

Ceph是能提供对象存储、块存储、共享文件系统多种存储形式,这里使用块存储,兼容性更好,灵活性更高。

# go to the ceph csi rbd folder
> cd [ur-root/path]/rook/cluster/examples/kubernetes/ceph/csi/rbd

# create ceph rdb storageclass for test
> kubectl apply -f storageclass-test.yaml
cephblockpool.ceph.rook.io/replicapool created
storageclass.storage.k8s.io/rook-ceph-block created

> kubectl get storageclass
NAME              PROVISIONER                  RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-ceph-block   rook-ceph.rbd.csi.ceph.com   Delete          Immediate           true                   4m7s

有了Ceph StorageClass,我们只需要申明PVC,就可以快速按需创建出一个块设备以及对应的PV,相比传统的需要手动首先创建PV,然后在声明对应的PVC,操作更简单,管理更方便。

下面是一个基于Ceph StorageClass的PVC yaml例子:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-pv-claim
spec:
  storageClassName: rook-ceph-block # this is created sc
  accessModes:
    - ReadWriteOnce # pay attention to this value
  resources:
    requests:
      storage: 1Gi

部署PVC,并观察PV是否自动创建:

> kubectl apply -f pvc.yaml
persistentvolumeclaim/ceph-pv-claim created

> kubectl get pvc
NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
ceph-pv-claim   Bound    pvc-fb2d6d97-d7aa-43df-808c-81f15e7a2797   1Gi        RWO            rook-ceph-block   3s

> kubectl get pvc,pv
NAME                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
persistentvolumeclaim/ceph-pv-claim   Bound    pvc-fb2d6d97-d7aa-43df-808c-81f15e7a2797   1Gi        RWO            rook-ceph-block   9s

# the pv has been created automatically and bound to the pvc we created
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS      REASON   AGE
persistentvolume/pvc-fb2d6d97-d7aa-43df-808c-81f15e7a2797   1Gi        RWO            Delete           Bound    default/ceph-pv-claim   rook-ceph-block            7s

创建一个基于PVC的Pod:

apiVersion: v1
kind: Pod
metadata:
  name: ceph-pv-pod
spec:
  volumes:
    - name: ceph-pv-storage
      persistentVolumeClaim:
        claimName: ceph-pv-claim # use the pvc we created
  containers:
    - name: nginx
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html" # mount the pv
          name: ceph-pv-storage

等待Pod部署完成,观察pod的存储挂载情况:

> kubectl exec -ti ceph-pv-pod -- bash
> df -h
Filesystem               Size  Used Avail Use% Mounted on
overlay                   39G  4.1G   35G  11% /
tmpfs                     64M     0   64M   0% /dev
tmpfs                    1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mapper/centos-root   17G  7.3G  9.8G  43% /etc/hosts
/dev/mapper/centos-data   39G  4.1G   35G  11% /etc/hostname
shm                       64M     0   64M   0% /dev/shm
# the block storage is mount to the pod
/dev/rbd0                976M  2.6M  958M   1% /usr/share/nginx/html
#
tmpfs                    1.9G   12K  1.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                    1.9G     0  1.9G   0% /proc/acpi
tmpfs                    1.9G     0  1.9G   0% /proc/scsi
tmpfs                    1.9G     0  1.9G   0% /sys/firmware

> cp -R /etc/ /usr/share/nginx/html/ # make some data into block storage

通过Ceph toolbox观察Ceph集群的使用情况:

> kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') -- bash 
> ceph status
  cluster:
    id:     cda62855-74ec-4aee-9037-26445f2538ba
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum a (age 11h)
    mgr: a(active, since 11h)
    osd: 3 osds: 3 up (since 10h), 3 in (since 10h)

  data:
    pools:   1 pools, 8 pgs
    objects: 20 objects, 21 MiB
    usage:   21 GiB used, 30 GiB / 51 GiB avail
    pgs:     8 active+clean

# we have some io load now
  io:
    client:   56 KiB/s wr, 0 op/s rd, 10 op/s wr

> ceph osd status
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
| id |  host  |  used | avail | wr ops | wr data | rd ops | rd data |   state   |
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
| 0  | node02 | 7336M | 9.82G |    0   |     0   |    0   |     0   | exists,up |
| 1  | node03 | 7447M | 9946M |    0   |     0   |    0   |     0   | exists,up |
| 2  | node01 | 6871M | 10.2G |    0   |     0   |    0   |   819   | exists,up |
+----+--------+-------+-------+--------+---------+--------+---------+-----------+

理解Access Mode属性

存储系统的访问安全控制在Kubernetes的时代得到了长足的进步,远远胜于纯Docker时代的简单粗暴。来看下Kubernetes在管理存储(PV、PVC)时提供了哪些访问控制机制:

  • RWO: ReadWriteOnce,只有单个节点可以挂载这个volume,进行读写操作;
  • ROX: ReadOnlyMany,多个节点可以挂载这个volume,只能进行读操作;
  • RWX: ReadWriteMany,多个节点可以挂载这个volume,读写操作都是允许的。

所以RWO、ROX和RWX只跟同时使用volume的worker节点数量有关,而不是跟pod数量!

以前苦于没有部署云原生存储系统,一直没法实践这些特性,这次得益于Rook的便捷性,赶紧来尝鲜下。计划测试两个场景:

  • 测试ReadWriteOnce,测试步骤如下:
    1. 首先部署一个使用ReadWriteOnce访问权限的PVC的名为ceph-pv-pod的单个pod实例
    2. 然后部署一个使用相同PVC的名为n2的deployment,1个pod实例
    3. 扩容n2至6个pod副本
    4. 观察结果
> kubectl get pod -o wide --sort-by=.spec.nodeName | grep -E '^(n2|ceph)'
NAME                  READY   STATUS              IP                NODE  
n2-7db787d7f4-ww2fp   0/1     ContainerCreating   <none>            node01
n2-7db787d7f4-8r4n4   0/1     ContainerCreating   <none>            node02
n2-7db787d7f4-q5msc   0/1     ContainerCreating   <none>            node02
n2-7db787d7f4-2pfvd   1/1     Running             100.96.174.137    node03
n2-7db787d7f4-c8r8k   1/1     Running             100.96.174.139    node03
n2-7db787d7f4-hrwv4   1/1     Running             100.96.174.138    node03
ceph-pv-pod           1/1     Running             100.96.174.135    node03

从上面的结果可以看到,由于ceph-pv-pod这个Pod优先绑定了声明为ReadWriteOnce的PVC,它所在的节点node03就能成功部署n2的pod实例,而调度到其他节点的n2就无法成功部署了,挑个看看错误信息:

> kubectl describe pod n2-7db787d7f4-ww2fp
...
Events:
  Type     Reason              Age                  From                     Message
  ----     ------              ----                 ----                     -------
  Normal   Scheduled           <unknown>            default-scheduler        Successfully assigned default/n2-7db787d7f4-ww2fp to node01
  Warning  FailedAttachVolume  10m                  attachdetach-controller  Multi-Attach error for volume "pvc-fb2d6d97-d7aa-43df-808c-81f15e7a2797" Volume is already used by pod(s) n2-7db787d7f4-c8r8k, ceph-pv-pod, n2-7db787d7f4-2pfvd, n2-7db787d7f4-hrwv4

从Pod Events中可以明显看到错误了,由于ReadWriteOnce的存在,无法使用Multi-Attach了,符合期待。

  • 测试ReadWriteMany,测试步骤如下:
    1. 首先部署一个使用 ReadWriteMany访问权限的PVC的名为2ceph-pv-pod的单个pod实例
    2. 然后部署一个使用相同PVC的名为n3的deployment,1个pod实例
    3. 扩容n3至6个pod副本
    4. 观察结果

原来是想直接改第一个测试场景的创建pvc的yaml,发现如下错误。意思是创建好的pvc除了申请的存储空间以外,其他属性是无法修改的。

> k apply -f pvc.yaml 
The PersistentVolumeClaim "ceph-pv-claim" is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims

只能重新创建了。。。但当声明创建新的PVC时,又发生了问题,pvc一直处于pending状态。。。

> kubectl get pvc
NAME            STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
ceph-pv-claim   Bound     pvc-fb2d6d97-d7aa-43df-808c-81f15e7a2797   1Gi        RWO            rook-ceph-block   36h
ceph-pvc-2      Pending                                                                        rook-ceph-block   10m
> kubectl describe pvc ceph-pvc-2
...
Events:
  Type     Reason                Age                   From                                                                                                        Message
  ----     ------                ----                  ----                                                                                                        -------
  Normal   Provisioning          4m41s (x11 over 13m)  rook-ceph.rbd.csi.ceph.com_csi-rbdplugin-provisioner-66f64ff49c-wvpkg_b78217fb-8739-4ced-9e18-7430fdde964b  External provisioner is provisioning volume for claim "default/ceph-pvc-2"
  Warning  ProvisioningFailed    4m41s (x11 over 13m)  rook-ceph.rbd.csi.ceph.com_csi-rbdplugin-provisioner-66f64ff49c-wvpkg_b78217fb-8739-4ced-9e18-7430fdde964b  failed to provision volume with StorageClass "rook-ceph-block": rpc error: code = InvalidArgument desc = multi node access modes are only supported on rbd `block` type volumes
  Normal   ExternalProvisioning  3m4s (x42 over 13m)   persistentvolume-controller                                                                                 waiting for a volume to be created, either by external provisioner "rook-ceph.rbd.csi.ceph.com" or manually created by system administrator

查看event详细后,发现了这个错误信息:

failed to provision volume with StorageClass "rook-ceph-block": rpc error: code = InvalidArgument desc = multi node access modes are only supported on rbd `block` type volumes

翻译过来的意思是:多节点访问模式只支持在rbd block类型的volume上配置。。。难道说ceph的这个rbd storageclass是个假的“块存储”。。。

一般发生这种不所措的错误,首先可以去官方Github的issue或pr里找找有没有类似的问题。经过一番搜索,找到一个maintainer的相关说法。如下图所示。意思是不推荐在ceph rbd模式下使用RWX访问控制,如果应用层没有访问锁机制,可能会造成数据损坏。

进而找到了官方上的说法

There are two CSI drivers integrated with Rook that will enable different scenarios:

  • RBD: This driver is optimized for RWO pod access where only one pod may access the storage
  • CephFS: This driver allows for RWX with one or more pods accessing the same storage

好吧,原来官方网站已经说明了CephFS模式是使用RWX模式的正确选择。

使用CephFS测试ReadWriteMany(RWX)模式

官方已经提供了支持CephFS的StorageClass,我们需要部署开启:

> cd [ur-rook-git]/rook/cluster/examples/kubernetes/ceph/csi/cephfs/
> kubectl apply -f storageclass.yaml
> kubectl get sc
NAME              PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION
csi-cephfs        rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                
rook-ceph-block   rook-ceph.rbd.csi.ceph.com      Delete          Immediate           true                

注意观察上面的StorageClass文件parameters里有一行:

# CephFS filesystem name into which the volume shall be created
  fsName: myfs

这是使用CephFS必须要有的文件系统,必须创建好,才能声明创建基于CephFS的PV和PVC。官方提供了示例:

> cat cephfs-filesystem.yaml
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: myfs # the name should be the same as the one in storageclass yaml
  namespace: rook-ceph
spec:
  metadataPool:
    replicated:
      size: 3
  dataPools:
    - replicated:
        size: 3
  preservePoolsOnDelete: true
  metadataServer:
    activeCount: 1
    activeStandby: true

> kubectl apply -f cephfs-filesystem.yaml

创建完CephFS的StorageClass和FileSystem,就可以测试了。测试场景为部署一个deployment,6个副本,使用RWX模式的Volume:

# pvc-cephfs.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cephfs-pvc
spec:
  storageClassName: csi-cephfs # new storageclass
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
---
# deployment-cephfs-pvc.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: n4-cephfs
    pv: cephfs
  name: n4-cephfs
spec:
  replicas: 3
  selector:
    matchLabels:
      app: n4-cephfs
      pv: cephfs
  template:
    metadata:
      labels:
        app: n4-cephfs
        pv: cephfs
    spec:
      volumes:
        - name: fsceph-pv-storage
          persistentVolumeClaim:
            claimName: cephfs-pvc
      containers:
      - image: nginx
        name: nginx
        ports:
          - containerPort: 80
            name: "http-server"
        volumeMounts:
          - mountPath: "/usr/share/nginx/html"
            name: fsceph-pv-storage

部署后观察每个pod的运行情况以及PV和PVC创建情况:

> kubectl get pod -l pv=cephfs -o wide
NAME                         READY   STATUS     IP                NODE  
n4-cephfs-859b956b65-768z7   1/1     Running    100.95.185.225    node02
n4-cephfs-859b956b65-cs9jn   1/1     Running    100.96.174.143    node03
n4-cephfs-859b956b65-rwdn4   1/1     Running    100.117.144.146   node01
n4-cephfs-859b956b65-x55s4   1/1     Running    100.96.174.142    node03
n4-cephfs-859b956b65-xffwb   1/1     Running    100.117.144.148   node01
n4-cephfs-859b956b65-xx8nz   1/1     Running    100.117.144.147   node01

> 
> kubectl get pvc,pv
NAME                                  STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
persistentvolumeclaim/cephfs-pvc      Bound     pvc-75b40dd7-b880-4d67-9da6-88aba8616466   1Gi        RWX            csi-cephfs

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS
persistentvolume/pvc-75b40dd7-b880-4d67-9da6-88aba8616466   1Gi        RWX            Delete           Bound    default/cephfs-pvc      csi-cephfs  

分布在不同的节点上的pod都能部署成功,PV也能创建绑定成功。符合测试预期。

更深入地观察存储挂载机制

通过上面两个测试场景,我们来看下背后的云原生存储的运行逻辑:

  • 进入pod观察存储挂载情况

对比两个测试场景pod实例里面存储挂载情况:

# use ceph as rbd storage
# it is mount as block device 
> df -h
/dev/rbd0                976M  3.3M  957M   1% /usr/share/nginx/html

# use ceph as file system storage
# it is mount as nfs storage
> df -h 
10.109.80.220:6789:/volumes/csi/csi-vol-1dc92634-79cd-11ea-96a3-26ab72958ea2  1.0G     0  1.0G   0% /usr/share/nginx/html

可以看到Ceph rbd和CephFS挂载到Pod里的方式是有差别的。

  • 观察主机层存储挂载情况
# use ceph as rbd storage
> df -h |grep rbd # on work node
/dev/rbd0                                                                     976M  3.3M  957M   1% /var/lib/kubelet/pods/e432e18d-b18f-4b26-8128-0b0219a60662/volumes/kubernetes.io~csi/pvc-fb2d6d97-d7aa-43df-808c-81f15e7a2797/mount

# use ceph as file system storage
> df -h |grep csi
10.109.80.220:6789:/volumes/csi/csi-vol-1dc92634-79cd-11ea-96a3-26ab72958ea2  1.0G     0  1.0G   0% /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-75b40dd7-b880-4d67-9da6-88aba8616466/globalmount

简单解释下主机层相关路径的命名规则:

/var/lib/kubelet/pods/<Pod的ID>/volumes/kubernetes.io~<Volume类型>/<Volume名字> 

最终都是通过docker run映射到容器里去:

docker run -v /var/lib/kubelet/pods/<Pod-ID>/volumes/kubernetes.io~<Volume类型>/<Volume名字>:/<容器内目标目录> 镜像 ...
  • 从Kubernetes观察存储挂载情况

Kubernetes提供了获取StorageClass、PV和Node之间的关系——volumeattachment资源类型。它的官方解释是:

VolumeAttachment captures the intent to attach or detach the specified volume to/from the specified node. VolumeAttachment objects are non-namespaced.

来看下当前的情况:

> kubectl get volumeattachment
NAME                                                                   ATTACHER                        PV                                         NODE     ATTACHED   AGE
csi-0453435f600da9580d6dff3bceb3d151d36462fa001c682663dd7371beff309f   rook-ceph.cephfs.csi.ceph.com   pvc-75b40dd7-b880-4d67-9da6-88aba8616466   node02   true       13h
csi-89e8c56d49e524d5d8dd6bfe4809f040f35f8b416a09241bcb9e6632ced232be   rook-ceph.cephfs.csi.ceph.com   pvc-75b40dd7-b880-4d67-9da6-88aba8616466   node03   true       13h
csi-97bab907614b96ddf2df6ce15d547e409b68bb41591f1b4bd25eacb9a59ec9de   rook-ceph.rbd.csi.ceph.com      pvc-fb2d6d97-d7aa-43df-808c-81f15e7a2797   node03   true       2d3h
csi-f002b45a7930af0b144fd0d40734ec3a41a207b892db471cb82d373b9bfd79bd   rook-ceph.cephfs.csi.ceph.com   pvc-75b40dd7-b880-4d67-9da6-88aba8616466   node01   true       13h

能看到每个主机层挂载点的详细情况,方便大家troubleshooting。

Ceph界面化管理Ceph Dashboard

Rook官方很贴心地提供了Ceph界面化管理的解决方案——Ceph dashboard。标准版部署Rook已经自带这个功能,默认是无法集群外访问的,手动expose为nodeport模式即可:

> kubectl -n rook-ceph get svc |grep dash
rook-ceph-mgr-dashboard                  ClusterIP   10.97.126.41     <none>        8443/TCP            2d14h
rook-ceph-mgr-dashboard-external-https   NodePort    10.109.153.31    <none>        8443:30010/TCP      14h 

通过浏览器访问https://node-ip:30010,默认登录用户名为admin,密码可以通过这样的方式获取:

kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo

登录后,界面如下。内容非常多,包括读写速率监控,健康监控等,绝对是Ceph管理的好帮手。

还提供交互式API文档,非常贴心。

戳视频可以看完整Demo:

总结

Rook能帮你快速搭建一套Production-Ready的云原生存储平台,同时提供全生命周期管理,适合初中高级全阶段的存储管理玩家。

本文涉及的部署物料可以去这里获取:

https://github.com/nevermosby/rook-ceph-tutorial

1 对 “Rook带你玩转云原生存储”的想法;

发表评论

电子邮件地址不会被公开。 必填项已用*标注