ceph手动部署

Linux
image-20241129094739776

一、 节点规划

主机名 IP地址 角色
ceph01.example.com 172.18.0.10/24 mon、mgr、osd、mds、rgw
ceph02.example.com 172.18.0.20/24 mon、mgr、osd、mds、rgw
ceph03.example.com 172.18.0.30/24 mon、mgr、osd、mds、rgw

操作系统版本: Rocky Linux release 8.8 (Green Obsidian)

二、 准备工作

以下操作所有主机都要操作

  1. 替换YUM源

sed -e ‘s|^mirrorlist=|#mirrorlist=|g’

-e ‘s|^#baseurl=http://dl.rockylinux.org/$contentdir|baseurl=https://mirrors.aliyun.com/rockylinux|g

-i.bak

/etc/yum.repos.d/Rocky-.repo

dnf makecache

  1. 关闭防火墙和SELINUX

systemctl disable –now firewalld.service

setenforce 0

vim /etc/sysconfig/selinux

  1. 配置时间同步

echo “server ntp.aliyun.com iburst” >> /etc/chronyd.conf

systemctl enable –now chronyd

chronyc sources

  1. 主机名与IP进行映射

vim /etc/hosts

172.18.0.10 ceph01.example.com ceph01

172.18.0.20 ceph02.example.com ceph02

172.18.0.30 ceph03.example.com ceph03

  1. 节点安装ceph软件包

yum install centos-release-ceph-pacific -y 此处使用P版 目前企业级存储使用的稳定版

yum install ceph -y

三、配置MON

  1. 生成集群UUID

uuidgen

eb6d6dab-6854-4663-ba7f-eadf2758e681

  1. 创建ceph.conf配置文件

vim /etc/ceph/ceph.conf

[global]

fsid=eb6d6dab-6854-4663-ba7f-eadf2758e681

mon_initial_members=ceph01,ceph02,ceph03

mon_host=172.18.0.10,172.18.0.20,172.18.0.30

public_network=172.18.0.0/24

auth_cluster_required=cephx

auth_service_required=cephx

auth_client_required=cephx

osd_pool_default_size=3

osd_pool_default_min_size=2

osd_pool_default_pg_num=128

osd_pool_default_pgp_num=128

storage_type=bluestore

osd_objectstore=bluestore

osd_mkfs_type=xfs

bluestore_block_size=5628755968

  1. 生成monitor keyring

ceph-authtool –create-keyring /tmp/ceph.mon.keyring –gen-key -n mon. –cap mon ‘allow ‘

  1. 生成admin管理员key

ceph-authtool –create-keyring /etc/ceph/ceph.client.admin.keyring –gen-key -n client.admin –cap mon ‘allow ‘ –cap osd ‘allow ‘ –cap mds ‘allow ‘ –cap mgr ‘allow ‘

  1. 生成OSD引导key

ceph-authtool –create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring –gen-key -n client.bootstrap-osd –cap mon ‘profile bootstrap-osd’ –cap mgr ‘allow r’

  1. 将admin和osd引导key导入到mon key中

ceph-authtool /tmp/ceph.mon.keyring –import-keyring /etc/ceph/ceph.client.admin.keyring

ceph-authtool /tmp/ceph.mon.keyring –import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring

chown ceph:ceph /tmp/ceph.mon.keyring

  1. 生成monmap映射视图

monmaptool –create –add ceph01 172.18.0.10 –add ceph02 172.18.0.20 –add ceph03 172.18.0.30 –fsid eb6d6dab-6854-4663-ba7f-eadf2758e681 /tmp/monmap

  1. 拷贝文件到其他节点

scp /etc/ceph/ceph.conf root@ceph02:/etc/ceph/

scp /etc/ceph/ceph.conf root@ceph03:/etc/ceph/

scp /tmp/monmap root@ceph02:/tmp/

scp /tmp/monmap root@ceph03:/tmp/

scp /tmp/ceph.mon.keyring root@ceph02:/tmp/

scp /tmp/ceph.mon.keyring root@ceph03:/tmp/

scp /etc/ceph/ceph.client.admin.keyring root@ceph02:/etc/ceph/

scp /etc/ceph/ceph.client.admin.keyring root@ceph03:/etc/ceph/

  1. 从映射视图导入集群关系(在所有节点上操作)

sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph01(ceph01改成不同的主机名)

ceph-mon –mkfs -i ceph01 –monmap /tmp/monmap –keyring /tmp/ceph.mon.keyring(ceph01改成不同的主机名)

chown ceph.ceph -R /var/lib/ceph

  1. 启动mon服务并查看集群状态(在所有节点上操作)

systemctl enable –now ceph-mon@ceph01.service (ceph01改成不同的主机名)

ceph -s 查看集群状态

问题1:3 monitors have not enabled msgr2

ceph mon enable-msgr2 ()

​ 问题2:mons are allowing insecure global_id reclaim

​ ceph config set mon auth_allow_insecure_global_id_reclaim false(禁用不安全模式)

四、配置MGR

1-3以下所有节点均需要操作

  1. 创建mgr key存放目录

sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-$(hostname -s)

  1. 生成mgr key

ceph auth get-or-create mgr. $(hostname -s) mon ‘allow profile mgr’ osd ‘allow ‘ mds ‘allow ‘ > /var/lib/ceph/mgr/ceph-$(hostname -s)/keyring

  1. 启动mgr服务

systemctl enable –now ceph-mgr@ceph01.service (ceph01在不同节点上换成不同主机名)

  1. 配置dashboard图形化控制台

​ 4.0 安装dashboard模块并启用

​ yum install ceph-mgr-dashboard -y

​ 4.1 生成自签名证书

​ openssl req -new -nodes -x509 -subj "/O=IT/CN=ceph-mgr-dashboard" -days 3650

-keyout dashboard.key -out dashboard.crt -extensions v3_ca

 4.2 导入证书

​ ceph dashboard set-ssl-certificate -i dashboard.crt

​ ceph dashboard set-ssl-certificate-key -i dashboard.key

​ 4.3 配置地址和端口

​ ceph config set mgr mgr/dashboard/server_addr 0.0.0.0

​ ceph config set mgr mgr/dashboard/ssl_server_port 8443

​ 4.4 禁用rgw api校验自签名证书(如果启用rgw ssl则需要)

​ ceph dashboard set-rgw-api-ssl-verify false

​ 4.5初始化 dashboard 用户

​ echo redhat123 > pass.txt

​ ceph dashboard ac-user-create admin administrator -i pass.txt

​ 4.6 启用http关闭ssl

​ mgr-dashboard默认使用ssl进行访问,关闭ssl可以使用http

​ ceph config set mgr mgr/dashboard/ssl false

​ 4.7 启用或禁用dashboard

​ ceph config module disable dashboard

​ ceph config module enable dashboard

五、配置OSD

  1. 拷贝OSD的密钥环(ceph01操作)

scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph02:/var/lib/ceph/bootstrap-osd/

scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph03:/var/lib/ceph/bootstrap-osd/

拷贝后修改文件的权限chown ceph.ceph /var/lib/ceph/bootstrap-osd/ceph.keyring

  1. 创建OSD

ceph-volume lvm create –data /dev/sdb –bluestore

说明:

–bluestore 不接时默认为bluestore后端驱动,使用–filestore则使用文件级filestore驱动,cephadm不支持管理后端驱动为filestore的OSD

​ 上述过程可以拆分成两步:

​ 准备OSD:ceph-volume lvm prepare –data /dev/sdb

​ 激活OSD:ceph-volume lvm activate {ID} {FSID}

​  ceph-volume lvm list 查看OSD的ID

  1. 查看集群状态以及添加的OSD

ceph -s 查看集群状态是否ok

ceph osd ls

六、配置MDS

​ 以下操作所有节点都要做

​ 1. 创建mds key存放目录

​ sudo -u ceph mkdir /var/lib/ceph/mds/ceph-$(hostname -s)

  1. 生成mds keyring

    ceph auth get-or-create mds.$(hostname -s) osd "allow rwx" mds "allow" mon "allow profile mds" > /var/lib/ceph/mds/ceph-$(hostname -s)/keyring

    chown ceph.ceph -R /var/lib/ceph/mds/ceph-ceph02/keyring

  2. 启动mds 服务

​ systemctl start ceph-mds@ceph01.service

​ systemctl start ceph-mds@ceph02.service

​ systemctl start ceph-mds@ceph03.service

  1. 创建文件系统测试

ceph osd pool create fspool_meta 创建文件系统元数据池

ceph osd pool create fspool_data 创建文件系统数据池

ceph fs new fs01 fspool_meta fspool_data 创建fs01的文件系统

ceph fs status 查看文件系统的主从节点

ceph fs authorize fs01 client.wangwu / rw 创建文件系统使用用户王五并授权

mount -t ceph ceph01:/ /media/ -o name=wangwu,fs=fs01,secret=AQB3ryJl1Z2QMRAAH1C/qRaBGIKKBWET5lM1eg== 挂载测试

七、配置RGW

1.安装软件包(所有节点操作)

​ yum install ceph-radosgw -y

2.创建 rgw key(ceph01操作)

​ ceph auth get-or-create client.rgw.ceph01 osd ‘allow rwx’ mon ‘allow rwx’ >> /etc/ceph/ceph.client.radosgw.keyring

​ ceph auth get-or-create client.rgw.ceph02 osd ‘allow rwx’ mon ‘allow rwx’ >> /etc/ceph/ceph.client.radosgw.keyring

​ ceph auth get-or-create client.rgw.ceph03 osd ‘allow rwx’ mon ‘allow rwx’ >> /etc/ceph/ceph.client.radosgw.keyring

​ chown ceph:ceph /etc/ceph/ceph.client.radosgw.keyring

​ scp /etc/ceph/ceph.client.radosgw.keyring root@ceph02:/etc/ceph/

​ chown ceph:ceph /etc/ceph/ceph.client.radosgw.keyring (ceph02操作)

​ scp /etc/ceph/ceph.client.radosgw.keyring root@ceph03:/etc/ceph/

​ chown ceph:ceph /etc/ceph/ceph.client.radosgw.keyring (ceph03操作)

4.编辑配置文件(ceph01操作)

vim /etc/ceph.conf 新增rgw配置

[client.rgw.ceph01]

host=ceph01

keyring=/etc/ceph/ceph.client.radosgw.keyring

log file=/var/log/radosgw/client.radosgw.gateway.log

rgw frontends = civetweb port=172.18.0.10:8080

[client.rgw.ceph02]

host=ceph02

keyring=/etc/ceph/ceph.client.radosgw.keyring

log file=/var/log/radosgw/client.radosgw.gateway.log

rgw frontends = civetweb port=172.18.0.20:8080

[client.rgw.ceph03]

host=ceph03

keyring=/etc/ceph/ceph.client.radosgw.keyring

log file=/var/log/radosgw/client.radosgw.gateway.log

rgw frontends = civetweb port=172.18.0.30:8080

scp /etc/ceph/ceph.conf root@ceph02:/etc/ceph/

scp /etc/ceph/ceph.conf root@ceph03:/etc/ceph/

  1. 创建日志目录(所有节点操作)

sudo -u ceph mkdir /var/log/radosgw/

  1. 启动RGW服务(对应节点操作)

systemctl enable –now ceph-radosgw@rgw.ceph01.service

systemctl enable –now ceph-radosgw@rgw.ceph02.service

systemctl enable –now ceph-radosgw@rgw.ceph03.service

服务启动后rgw将自动创建以下存储池

.rgw.root #包含 realm(领域信息),比如 zone 和 zonegroup

default.rgw.log #存储日志信息,用于记录各种 log 信息

default.rgw.control #系统控制池,在有数据更新时,通知其它 RGW 更新缓存

default.rgw.meta #元数据存储池,通过不同的名称空间分别存储不同的 rados 对象

  1. 创建s3用户测试

radosgw-admin user create –uid=user1 –access-key=123 –secret=456 –email user1@example.com –display-name user1

  1. s3对象存储测试(任意选择一个节点测试)

    7.1 安装s3工具

​ yum install epel-release -y 使用epel源

​ yum install aws -y 安装s3客户端工具

​ 7.2 配置s3访问凭据

​ aws configure –profile=ceph 凭据名为ceph

​ 输入ak和sk

7.3 创建存储桶

​ aws –profile=ceph –endpoint-url=http://172.18.0.10:8080 s3 mb s3://bucket1

7.4 上传文件到桶

​ aws –profile=ceph –endpoint-url http://172.18.0.10:8080 s3 cp /etc/passwd s3://bucket1/passwd

本文链接:https://www.orcy.net.cn/4040.html,转载请附上原文出处链接

暂无评论

发表评论

电子邮件地址不会被公开。 必填项已用*标注