- ceph-users - lists.ceph.io

Spam from Chip Cox
by Frank Schilder 03 May '21

03 May '21

Does anyone else receive unsolicited replies from sender "Chip Cox <chip(a)softiron.com>" to e-mails posted on this list? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

[ Ceph MDS MON Config Variables ] Failover Delay issue
by Lokendra Rathour 03 May '21

03 May '21

Hi Team, I was setting up the ceph cluster with - Node Details:3 Mon,2 MDS, 2 Mgr, 2 RGW - Deployment Type: Active Standby - Testing Mode: Failover of MDS Node - Setup : Octopus (15.2.7) - OS: centos 8.3 - hardware: HP - Ram: 128 GB on each Node - OSD: 2 ( 1 tb each) - Operation: Normal I/O with mkdir on every 1 second. T*est Case: Power-off any active MDS Node for failover to happen* *Observation:* We have observed that whenever an active MDS Node is down it takes around* 40 seconds* to activate the standby MDS Node. on further checking the logs for the new-handover MDS Node we have seen delay on the basis of following inputs: 1. 10 second delay after which Mon calls for new Monitor election 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 calling monitor election 2. 5 second delay in which newly elected Monitor is elected 1. [log] 0 log_channel(cluster) log [INF] : mon.cephnode1 is new leader, mons cephnode1,cephnode3 in quorum (ranks 0,2) 3. the addition beacon grace time for which the system waits before which it enables standby MDS node activation. (approx delay of 19 seconds) 1. defaults : sudo ceph config get mon mds_beacon_grace 15.000000 2. sudo ceph config get mon mds_beacon_interval 5.000000 3. [log] - 2021-04-30T18:23:10.136+0530 7f4e3925c700 1 mon.cephnode2(a)1(leader).mds e776 no beacon from mds.0.771 (gid: 639443 addr: [v2: 10.0.4.10:6800/2172152716,v1:10.0.4.10:6801/2172152716] state: up:active)* since 18.7951* 4. *in Total it takes around 40 seconds to handover and activate passive standby node. * *Query:* 1. Can these variables be configured ? which we have tried,but are not aware of the overall impact on the ceph cluster because of these changes 1. By tuning these values we could reach the minimum time of 12 seconds in which the active node comes up. 2. Values taken to get the said time : 1. *mon_election_timeout* (default 5) - configured as 1 2. *mon_lease*(default 5) - configured as 2 3. *mds_beacon_grace* (default 15) - configured as 5 4. *mds_beacon_interval* (default 5) - configured as 1 We need to tune this setup to get the failover duration as low as 5-7 seconds. Please suggest/support and share your inputs, my setup is ready and already we are testing with multiple scenarios so that we are able to achive min failover duration. -- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

5 11

How can I get tail information a parted rados object
by by morphin 03 May '21

03 May '21

Hello. I'm trying to export objects from rados with rados get. Some objects bigger than 4M and they have tails. Is there any easy way to get tail information an object? For example this is an object: - c106b26b.3_Img/2017/12/im034113.jpg These are the objet parts: - c106b26b.3__multipart_Img/2017/12/im034113.jpg.2~fjrC5r_KCWMBat_4bFVtmBv9pxcVL-9.1 - c106b26b.3__shadow_Img/2017/12/im034113.jpg.2~fjrC5r_KCWMBat_4bFVtmBv9pxcVL-9.1_1 - c106b26b.3__shadow_Img/2017/12/im034113.jpg.2~fjrC5r_KCWMBat_4bFVtmBv9pxcVL-9.1_2 - c106b26b.3__multipart_Img/2017/12/im034113.jpg.2~fjrC5r_KCWMBat_4bFVtmBv9pxcVL-9.2 As you can see the object has 2 multipart and 2 shadow object. This jpg only works when I get all the parts and make it one with the order. order: "cat 9.1 9.1_1 9.1_2 9.2 > im034113.jpg" I'm trying to write a code and the code gonna read objects from a list and find all the parts, bring it together with the order... But I couldn't find a good way to get part information. I followed the link https://www.programmersought.com/article/31497869978/ and I get the object manifest with getxattr and decode it with "ceph-dencoder type RGWBucketEnt decode dump_json" But in the manifest I can not find a path to code it. It's not useful. Is there any different place that I can take the part information an object? Or better! Is there any tool to export an object with its tails? btw: these objects created by RGW using s3. RGW can not access these files. Because of that I'm trying to export it from rados and send it to different RGW.

2 2

Troubleshoot MDS failure
by Alessandro Piazza 03 May '21

03 May '21

Dear all, I'm having a hard time troubleshooting a file-system failure on my 3 node cluster (deployed with cephadm + docker). After moving some files between folders, the cluster became laggy and Metadata Servers started failing and got stuck in rejoin state. Of course I already tried to restart the cluster multiple times. The mds units are now in a failed state because of too many restarts, the file-system is degraded and cannot be mounted because no mds is up. I think the data pool is ok because I can get files using rados. I can trigger the standby mds to become the "major" with ceph orch daemon rm mds <mds-in-error-id> or deploy a new one but the new "major" mds go again in error state. I don't find the mds logs really helpful but you can find one in the attachments for someone more expert than me. I am hesitant to follow the guide https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ because of the warnings and because the ceph-journal-tool is poorly documented. The following might be useful seppia:~# ceph fs status starfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 rejoin(laggy) starfs.polposition.njarir 539 25 17 0 POOL TYPE USED AVAIL cephfs.starfs.meta metadata 9900M 1027G cephfs.starfs.data data 12.1T 1027G MDS version: ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable) seppia:~ # ceph health detail HEALTH_WARN 2 failed cephadm daemon(s); 1 filesystem is degraded; insufficient standby MDS daemons available; 7 pgs not deep-scrubbed in time [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon mds.starfs.polposition.njarir on polposition.starfleet.sns.it is in error state daemon mds.starfs.seppia.wdwrho on seppia.starfleet.sns.it is in error state [WRN] FS_DEGRADED: 1 filesystem is degraded fs starfs is degraded [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more [WRN] PG_NOT_DEEP_SCRUBBED: 7 pgs not deep-scrubbed in time pg 3.a8 not deep-scrubbed since 2021-04-20T20:07:48.346677+0000 pg 3.a2 not deep-scrubbed since 2021-04-21T08:10:55.220263+0000 pg 3.7 not deep-scrubbed since 2021-04-21T07:24:20.073569+0000 pg 2.0 not deep-scrubbed since 2021-04-21T05:01:18.439456+0000 pg 9.1a not deep-scrubbed since 2021-04-21T05:18:20.171151+0000 pg 3.1cb not deep-scrubbed since 2021-04-20T21:54:38.251349+0000 pg 3.1ef not deep-scrubbed since 2021-04-21T07:07:18.842132+0000 Thanks for any suggestions, Alessandro Piazza

1 0

Re: cephfs mount problems with 5.11 kernel - not a ipv6 problem
by Ilya Dryomov 03 May '21

03 May '21

On Mon, May 3, 2021 at 12:00 PM Magnus Harlander <magnus(a)harlan.de> wrote: > > Am 03.05.21 um 11:22 schrieb Ilya Dryomov: > > max_osd 12 > > I never had more then 10 osds on the two osd nodes of this cluster. > > I was running a 3 osd-node cluster earlier with more than 10 > osds, but the current cluster has been setup from scratch and > I definitely don't remember having ever more than 10 osds! > Very strange! > > I had to replace 2 disks because of DOA-Problems, but for that > I removed 2 osds and created new ones. > > I used ceph-deploy do create new osds. > > To delete osd.8 I used: > > # take it out > ceph osd out 8 > > # wait for rebalancing to finish > > systemctl stop ceph-osd@8 > > # wait for a healthy cluster > > ceph osd purge 8 --yes-i-really-mean-it > > # edit ceph.conf and remove osd.8 > > ceph-deploy --overwrie-conf admin s0 s1 > > # Add the new disk and: > ceph-deploy osd create --data /dev/sdc s0 > ... > > it get's created with the next free osd num (8) because purge releases 8 for reuse It would be nice to track it down, but for the immediate issue of kernel 5.11 not working, "ceph osd setmaxosd 10" should fix it. Thanks, Ilya

1 0

Cannot create issue in bugtracker
by Tobias Urdin 03 May '21

03 May '21

Hello, Is it only me that's getting Internal error when trying to create issues in the bugtracker for some day or two? https://tracker.ceph.com/issues/new Best regards

2 2

Re: cephfs mount problems with 5.11 kernel - not a ipv6 problem
by Ilya Dryomov 03 May '21

03 May '21

On Mon, May 3, 2021 at 9:20 AM Magnus Harlander <magnus(a)harlan.de> wrote: > > Am 03.05.21 um 00:44 schrieb Ilya Dryomov: > > On Sun, May 2, 2021 at 11:15 PM Magnus Harlander <magnus(a)harlan.de> wrote: > > Hi, > > I know there is a thread about problems with mounting cephfs with 5.11 kernels. > > ... > > Hi Magnus, > > What is the output of "ceph config dump"? > > Instead of providing those lines, can you run "ceph osd getmap 64281 -o > osdmap.64281" and attach osdmap.64281 file? > > Thanks, > > Ilya > > Hi Ilya, > > [root@s1 ~]# ceph config dump > WHO MASK LEVEL OPTION VALUE RO > global basic device_failure_prediction_mode local > global advanced ms_bind_ipv4 false > mon advanced auth_allow_insecure_global_id_reclaim false > mon advanced mon_lease 8.000000 > mgr advanced mgr/devicehealth/enable_monitoring true > > getmap output is attached, I see the problem, but I don't understand the root cause yet. It is related to the two missing OSDs: > May 02 22:54:05 islay kernel: libceph: no match of type 1 in addrvec > May 02 22:54:05 islay kernel: libceph: corrupt full osdmap (-2) epoch 64281 off 3154 (00000000a90fe1d7 of 000000000083f4bd-00000000c03bdc9b) > max_osd 12 > osd.0 up in ... [v2:192.168.200.141:6804/3027,v1:192.168.200.141:6805/3027] ... exists,up 631bc170-45fd-4948-9a5e-4c278569c0bc > osd.1 up in ... [v2:192.168.200.140:6811/3066,v1:192.168.200.140:6813/3066] ... exists,up 660a762c-001d-4160-a9ee-d0acd078e776 > osd.2 up in ... [v2:192.168.200.141:6815/3008,v1:192.168.200.141:6816/3008] ... exists,up e4d94d3a-ec58-46a1-b61c-c47dd39012ed > osd.3 up in ... [v2:192.168.200.140:6800/3067,v1:192.168.200.140:6801/3067] ... exists,up 26d25060-fd99-4d15-a1b2-ebb77646671e > osd.4 up in ... [v2:192.168.200.140:6804/3049,v1:192.168.200.140:6806/3049] ... exists,up 238f197d-ecbc-4588-8a99-6a63c9bb1a17 > osd.5 up in ... [v2:192.168.200.140:6816/3073,v1:192.168.200.140:6817/3073] ... exists,up a9dcb26f-0f1c-4067-a26b-a29939285e0b > osd.6 up in ... [v2:192.168.200.141:6808/3020,v1:192.168.200.141:6809/3020] ... exists,up f399b47d-063f-4b2f-bd93-289377dc9945 > osd.7 up in ... [v2:192.168.200.141:6800/3023,v1:192.168.200.141:6801/3023] ... exists,up 3557ceca-7bd8-401e-abd3-59bee168e8f6 > osd.8 up in ... [v2:192.168.200.141:6812/3017,v1:192.168.200.141:6813/3017] ... exists,up 7f9cad3f-163d-4bb7-85b2-fffd46982fff > osd.9 up in ... [v2:192.168.200.140:6805/3053,v1:192.168.200.140:6807/3053] ... exists,up c543b12a-f9bf-4b83-af16-f6b8a3926e69 The kernel client is failing to parse addrvec entries for non-existent osd10 and osd11. It is probably being too stringent, but before fixing it I'd like to understand what happened to those OSDs. It looks like they were removed but not completely. What let to their removal? What commands were used? Thanks, Ilya

1 0

global multipart lc policy in radosgw
by Boris Behrens 02 May '21

02 May '21

Hi, I have a lot of multipart uploads that look like they never finished. Some of them date back to 2019. Is there a way to clean them up when they didn't finish in 28 days? I know I can implement a LC policy per bucket, but how do I implement it cluster wide? Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 1

cephfs mount problems with 5.11 kernel - not a ipv6 problem
by Magnus Harlander 02 May '21

02 May '21

Hi, I know there is a thread about problems with mounting cephfs with 5.11 kernels. I tried everything that's mentioned there, but I still can not mount a cephfs from an octopus node. I verified: - I can not mount with 5.11 client kernels (fedora 33 and ubuntu 21.04) - I can mount with 5.10 client kernels - It is not due to ipv4/ipv6. I'm not using ipv6 - I'm using a cluster network on a private network segment. Because this was mentioned as a possible cause for the problems (next to ipv6) I removed the cluster network and now I'm using the same network for osd syncs and client connections. It did not help. - mount returns with a timeout and error after about 1 minute - I tried the ms_mode=legacy (and others) mount options. Nothing helped - I tried to use IP:PORT:/fs to mount to exclude DNS as the cause. Didn't help. - I did setup a similar test cluster on a few VMs and did not have a problem with mouting. Even used cluster networks, which also worked fine. I'm running out of ideas? Any help would be appreciated. \Magnus My Setup: SERVER OS: ========== [root@s1 ~]# hostnamectl Static hostname: s1.harlan.de Icon name: computer-desktop Chassis: desktop Machine ID: 3a0a6308630842ffad6b9bb8be4c7547 Boot ID: ffb2948d3934419dafceb0990316d9fd Operating System: CentOS Linux 8 CPE OS Name: cpe:/o:centos:centos:8 Kernel: Linux 4.18.0-240.22.1.el8_3.x86_64 Architecture: x86-64 CEPH VERSION: ============= ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable) CLIENT OS: ========== [root@islay ~]# hostnamectl Static hostname: islay Icon name: computer-laptop Chassis: laptop Machine ID: 6de7b27dfd864e9ea52b8b0cff47cdfc Boot ID: 6d8d8bb36f274458b2b761b0a046c8ad Operating System: Fedora 33 (Workstation Edition) CPE OS Name: cpe:/o:fedoraproject:fedora:33 Kernel: Linux 5.11.16-200.fc33.x86_64 Architecture: x86-64 CEPH VERSION: ============= [root@islay harlan]# ceph version ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable) [root@s1 ~]# ceph version ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable) FSTAB ENTRY: ============ cfs0,cfs1:/fs /data/fs ceph rw,_netdev,name=admin,secretfile=/etc/ceph/fs.secret 0 0 IP CONFIG MON/OSD NODE (s1) ======================= [root@s1 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp4s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000 link/ether 98:de:d0:04:26:86 brd ff:ff:ff:ff:ff:ff 3: enp5s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000 link/ether a8:a1:59:18:e7:ea brd ff:ff:ff:ff:ff:ff 4: vmbr: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 98:de:d0:04:26:86 brd ff:ff:ff:ff:ff:ff inet 192.168.200.111/24 brd 192.168.200.255 scope global noprefixroute vmbr valid_lft forever preferred_lft forever inet 192.168.200.141/24 brd 192.168.200.255 scope global secondary noprefixroute vmbr valid_lft forever preferred_lft forever inet 192.168.200.101/24 brd 192.168.200.255 scope global secondary vmbr valid_lft forever preferred_lft forever inet6 fe80::be55:705d:7c9e:eaa4/64 scope link noprefixroute valid_lft forever preferred_lft forever 5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr state UP group default qlen 1000 link/ether 98:de:d0:04:26:86 brd ff:ff:ff:ff:ff:ff 6: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 52:54:00:32:ea:2f brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 7: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000 link/ether 52:54:00:32:ea:2f brd ff:ff:ff:ff:ff:ff 8: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master vmbr state UNKNOWN group default qlen 1000 link/ether fe:54:00:67:4d:15 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe67:4d15/64 scope link valid_lft forever preferred_lft forever CEPH STATUS: ============ [root@s1 ~]# ceph -s cluster: id: 86bbd6c5-ae96-4c78-8a5e-50623f0ae524 health: HEALTH_OK services: mon: 4 daemons, quorum s0,mbox,s1,r1 (age 6h) mgr: s1(active, since 6h), standbys: s0 mds: fs:1 {0=s1=up:active} 1 up:standby osd: 10 osds: 10 up (since 6h), 10 in (since 6h) data: pools: 6 pools, 289 pgs objects: 1.75M objects, 1.6 TiB usage: 3.3 TiB used, 13 TiB / 16 TiB avail pgs: 289 active+clean io: client: 0 B/s rd, 245 KiB/s wr, 0 op/s rd, 4 op/s wr CEPH OSD TREE: ============== [root@s1 ~]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 16.99994 root default -9 8.39996 host s0 1 hdd 4.00000 osd.1 up 1.00000 1.00000 5 hdd 1.79999 osd.5 up 1.00000 1.00000 9 hdd 1.79999 osd.9 up 1.00000 1.00000 3 ssd 0.50000 osd.3 up 1.00000 1.00000 4 ssd 0.29999 osd.4 up 1.00000 1.00000 -12 8.59998 host s1 6 hdd 1.79999 osd.6 up 1.00000 1.00000 7 hdd 1.79999 osd.7 up 1.00000 1.00000 8 hdd 4.00000 osd.8 up 1.00000 1.00000 0 ssd 0.50000 osd.0 up 1.00000 1.00000 2 ssd 0.50000 osd.2 up 1.00000 1.00000 CEPH MON STAT: ============== [root@s1 ~]# ceph mon stat e19: 4 mons at {mbox=[v2:192.168.200.5:3300/0,v1:192.168.200.5:6789/0],r1=[v2:192.168.200.113:3300/0,v1:192.168.200.113:6789/0],s0=[v2:192.168.200.110:3300/0,v1:192.168.200.110:6789/0],s1=[v2:192.168.200.111:3300/0,v1:192.168.200.111:6789/0]}, election epoch 8618, leader 0 s0, quorum 0,1,2,3 s0,mbox,s1,r1 CEPH FS DUMP: ============= [root@s1 ~]# ceph fs dump dumped fsmap epoch 15534 e15534 enable_multiple, ever_enabled_multiple: 0,0 compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 2 Filesystem 'fs' (2) fs_name fs epoch 15534 flags 12 created 2021-02-02T18:47:25.306744+0100 modified 2021-05-02T16:33:36.738341+0200 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 min_compat_client 0 (unknown) last_failure 0 last_failure_osd_epoch 64252 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0 up {0=54782953} failed damaged stopped data_pools [10] metadata_pool 11 inline_data disabled balancer standby_count_wanted 1 [mds.s1{0:54782953} state up:active seq 816 addr [v2:192.168.200.111:6800/1895356761,v1:192.168.200.111:6801/1895356761]] Standby daemons: [mds.s0{-1:54958514} state up:standby seq 1 addr [v2:192.168.200.110:6800/297471268,v1:192.168.200.110:6801/297471268]] CEPH CONF: ========== [root@s1 ~]# cat /etc/ceph/ceph.conf [global] fsid = 86bbd6c5-ae96-4c78-8a5e-50623f0ae524 mon_initial_members = s0, s1, mbox, r1 mon_host = 192.168.200.110,192.168.200.111,192.168.200.5,192.168.200.113 ms_bind_ipv4 = true ms_bind_ipv6 = false auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public network = 192.168.200.0/24 [osd] public network = 192.168.200.0/24 osd_memory_target = 2147483648 osd crush update on start = false [osd.1] public addr = 192.168.200.140 osd_memory_target = 2147483648 [osd.3] public addr = 192.168.200.140 osd_memory_target = 2147483648 [osd.4] public addr = 192.168.200.140 osd_memory_target = 2147483648 [osd.5] public addr = 192.168.200.140 osd_memory_target = 2147483648 [osd.9] public addr = 192.168.200.140 osd_memory_target = 2147483648 [osd.0] public addr = 192.168.200.141 osd_memory_target = 2147483648 [osd.2] public addr = 192.168.200.141 osd_memory_target = 2147483648 [osd.6] public addr = 192.168.200.141 osd_memory_target = 2147483648 [osd.7] public addr = 192.168.200.141 osd_memory_target = 2147483648 [osd.8] public addr = 192.168.200.141 osd_memory_target = 2147483648 CEPH FS STAT ============ [root@s1 ~]# ceph fs status fs - 0 clients == RANK STATE MDS ACTIVITY DNS INOS 0 active s1 Reqs: 0 /s 0 0 POOL TYPE USED AVAIL cfs_md metadata 2365M 528G cfs data 2960G 4967G STANDBY MDS s0 VERSION DAEMONS None s1 ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable) s0 CLIENT JOURNALCTL WHEN MOUNTING =============================== May 02 22:54:04 islay kernel: FS-Cache: Loaded May 02 22:54:05 islay kernel: Key type ceph registered May 02 22:54:05 islay kernel: libceph: loaded (mon/osd proto 15/24) May 02 22:54:05 islay kernel: FS-Cache: Netfs 'ceph' registered for caching May 02 22:54:05 islay kernel: ceph: loaded (mds proto 32) May 02 22:54:05 islay kernel: libceph: mon1 (1)192.168.200.111:6789 session established May 02 22:54:05 islay kernel: libceph: mon1 (1)192.168.200.111:6789 socket closed (con state OPEN) May 02 22:54:05 islay kernel: libceph: mon1 (1)192.168.200.111:6789 session lost, hunting for new mon May 02 22:54:05 islay kernel: libceph: mon0 (1)192.168.200.5:6789 session established May 02 22:54:05 islay kernel: libceph: no match of type 1 in addrvec May 02 22:54:05 islay kernel: libceph: corrupt full osdmap (-2) epoch 64281 off 3154 (00000000a90fe1d7 of 000000000083f4bd-00000000c03bdc9b) May 02 22:54:05 islay kernel: osdmap: 00000000: 08 07 4f 24 00 00 09 01 9e 12 00 00 86 bb d6 c5 ..O$............ May 02 22:54:05 islay kernel: osdmap: 00000010: ae 96 4c 78 8a 5e 50 62 3f 0a e5 24 19 fb 00 00 ..Lx.^Pb?..$.... May 02 22:54:05 islay kernel: osdmap: 00000020: 54 f0 53 5d 3a fd ae 0e 1b 07 8f 60 b3 8e d2 2f T.S]:......`.../ May 02 22:54:05 islay kernel: osdmap: 00000030: 06 00 00 00 02 00 00 00 00 00 00 00 1d 05 44 01 ..............D. May 02 22:54:05 islay kernel: osdmap: 00000040: 00 00 01 02 02 02 20 00 00 00 20 00 00 00 00 00 ...... ... ..... May 02 22:54:05 islay kernel: osdmap: 00000050: 00 00 00 00 00 00 5e fa 00 00 2e 04 00 00 00 00 ......^......... May 02 22:54:05 islay kernel: osdmap: 00000060: 00 00 5e fa 00 00 00 00 00 00 00 00 00 00 00 00 ..^............. ..... many more lines, i can provide if they are useful. CEPH OSDMAP: ============ epoch 64281 fsid 86bbd6c5-ae96-4c78-8a5e-50623f0ae524 created 2019-08-14T13:28:20.246349+0200 modified 2021-05-02T22:10:03.802328+0200 flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 140 full_ratio 0.92 backfillfull_ratio 0.9 nearfull_ratio 0.88 require_min_compat_client jewel min_compat_client jewel require_osd_release octopus pool 2 'vms' replicated size 2 min_size 1 crush_rule 2 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 64094 lfor 0/62074/62072 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 8 'ssdpool' replicated size 2 min_size 1 crush_rule 2 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 61436 lfor 0/61436/61434 flags hashpspool stripe_width 0 pool 9 'hddpool' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 61413 lfor 0/61413/61411 flags hashpspool stripe_width 0 pool 10 'cfs' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 63328 flags hashpspool,selfmanaged_snaps stripe_width 0 application cephfs pool 11 'cfs_md' replicated size 2 min_size 1 crush_rule 2 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 63332 flags hashpspool stripe_width 0 application cephfs pool 12 'device_health_metrics' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 64255 flags hashpspool stripe_width 0 application mgr_devicehealth max_osd 12 osd.0 up in weight 1 up_from 64236 up_thru 64263 down_at 64233 last_clean_interval [64211,64231) [v2:192.168.200.141:6804/3027,v1:192.168.200.141:6805/3027] [v2:192.168.200.111:6806/3027,v1:192.168.200.111:6807/3027] exists,up 631bc170-45fd-4948-9a5e-4c278569c0bc osd.1 up in weight 1 up_from 64259 up_thru 64260 down_at 64249 last_clean_interval [64223,64248) [v2:192.168.200.140:6811/3066,v1:192.168.200.140:6813/3066] [v2:192.168.200.110:6813/3066,v1:192.168.200.110:6815/3066] exists,up 660a762c-001d-4160-a9ee-d0acd078e776 osd.2 up in weight 1 up_from 64236 up_thru 64266 down_at 64233 last_clean_interval [64211,64231) [v2:192.168.200.141:6815/3008,v1:192.168.200.141:6816/3008] [v2:192.168.200.111:6816/3008,v1:192.168.200.111:6817/3008] exists,up e4d94d3a-ec58-46a1-b61c-c47dd39012ed osd.3 up in weight 1 up_from 64256 up_thru 64264 down_at 64249 last_clean_interval [64221,64248) [v2:192.168.200.140:6800/3067,v1:192.168.200.140:6801/3067] [v2:192.168.200.110:6802/3067,v1:192.168.200.110:6803/3067] exists,up 26d25060-fd99-4d15-a1b2-ebb77646671e osd.4 up in weight 1 up_from 64256 up_thru 64264 down_at 64249 last_clean_interval [64221,64248) [v2:192.168.200.140:6804/3049,v1:192.168.200.140:6806/3049] [v2:192.168.200.110:6806/3049,v1:192.168.200.110:6807/3049] exists,up 238f197d-ecbc-4588-8a99-6a63c9bb1a17 osd.5 up in weight 1 up_from 64260 up_thru 64260 down_at 64249 last_clean_interval [64226,64248) [v2:192.168.200.140:6816/3073,v1:192.168.200.140:6817/3073] [v2:192.168.200.110:6818/3073,v1:192.168.200.110:6819/3073] exists,up a9dcb26f-0f1c-4067-a26b-a29939285e0b osd.6 up in weight 1 up_from 64240 up_thru 64260 down_at 64233 last_clean_interval [64218,64231) [v2:192.168.200.141:6808/3020,v1:192.168.200.141:6809/3020] [v2:192.168.200.111:6810/3020,v1:192.168.200.111:6811/3020] exists,up f399b47d-063f-4b2f-bd93-289377dc9945 osd.7 up in weight 1 up_from 64238 up_thru 64260 down_at 64233 last_clean_interval [64214,64231) [v2:192.168.200.141:6800/3023,v1:192.168.200.141:6801/3023] [v2:192.168.200.111:6802/3023,v1:192.168.200.111:6803/3023] exists,up 3557ceca-7bd8-401e-abd3-59bee168e8f6 osd.8 up in weight 1 up_from 64242 up_thru 64260 down_at 64233 last_clean_interval [64216,64231) [v2:192.168.200.141:6812/3017,v1:192.168.200.141:6813/3017] [v2:192.168.200.111:6814/3017,v1:192.168.200.111:6815/3017] exists,up 7f9cad3f-163d-4bb7-85b2-fffd46982fff osd.9 up in weight 1 up_from 64257 up_thru 64257 down_at 64249 last_clean_interval [64229,64248) [v2:192.168.200.140:6805/3053,v1:192.168.200.140:6807/3053] [v2:192.168.200.110:6808/3053,v1:192.168.200.110:6809/3053] exists,up c543b12a-f9bf-4b83-af16-f6b8a3926e69 blacklist 192.168.200.110:0/3803039218 expires 2021-05-03T15:33:52.837358+0200 blacklist 192.168.200.111:6800/3725740504 expires 2021-05-03T15:37:38.953040+0200 blacklist 192.168.200.110:6822/3464419 expires 2021-05-03T15:56:28.124585+0200 blacklist 192.168.200.110:6801/838484672 expires 2021-05-03T15:56:13.108594+0200 blacklist 192.168.200.110:6800/838484672 expires 2021-05-03T15:56:13.108594+0200 blacklist 192.168.200.111:6841/159804987 expires 2021-05-03T14:54:05.413130+0200 blacklist 192.168.200.111:6840/159804987 expires 2021-05-03T14:54:05.413130+0200 blacklist 192.168.200.111:6801/3725740504 expires 2021-05-03T15:37:38.953040+0200 blacklist 192.168.200.110:6807/453197 expires 2021-05-03T15:33:52.837358+0200 blacklist 192.168.200.5:6801/3078236863 expires 2021-05-03T14:38:57.694004+0200 blacklist 192.168.200.110:0/1948864559 expires 2021-05-03T15:33:52.837358+0200 blacklist 192.168.200.111:6800/3987205903 expires 2021-05-03T15:32:12.633802+0200 blacklist 192.168.200.111:6800/2342337613 expires 2021-05-03T14:46:57.936272+0200 blacklist 192.168.200.110:0/3020995128 expires 2021-05-03T15:56:28.124585+0200 blacklist 192.168.200.5:6800/3078236863 expires 2021-05-03T14:38:57.694004+0200 blacklist 192.168.200.110:0/2607867017 expires 2021-05-03T15:33:52.837358+0200 blacklist 192.168.200.111:6801/3987205903 expires 2021-05-03T15:32:12.633802+0200 blacklist 192.168.200.110:0/3159222459 expires 2021-05-03T15:56:28.124585+0200 blacklist 192.168.200.110:6806/453197 expires 2021-05-03T15:33:52.837358+0200 blacklist 192.168.200.110:6823/3464419 expires 2021-05-03T15:56:28.124585+0200 blacklist 192.168.200.111:6801/2342337613 expires 2021-05-03T14:46:57.936272+0200 blacklist 192.168.200.111:6800/2205788037 expires 2021-05-03T14:56:56.448631+0200 blacklist 192.168.200.111:6801/2205788037 expires 2021-05-03T14:56:56.448631+0200 -- Dr. Magnus Harlander Mail: harlan(a)harlan.de Web: www.harlan.de Stiftung: www.harlander-stiftung.de Ceterum censeo bitcoin esse delendam!

2 1

How radosgw works ?
by Fabrice Bacchella 02 May '21

02 May '21

I'm trying to understand what and where radosgw listen ? There is a lot of contradictory or redundant informations about that. First about the contradictory informations for the socket. At https://docs.ceph.com/en/pacific/radosgw/config-ref/ <https://docs.ceph.com/en/pacific/radosgw/config-ref/>, it says rgw_socket_path, but at https://docs.ceph.com/en/pacific/man/8/radosgw/ <https://docs.ceph.com/en/pacific/man/8/radosgw/> is says 'rgw socket path' That problem is quite common in the ceph documentation. Are both value accepted ? Next about some naming, or binding IP. Where it's defined, and how ? You have: rgw_frontends = "beast ssl_endpoint=0.0.0.0:443 port=443 ..." rgw_host = rgw_port = rgw_dns_name = That's a lot of redundancy, or contradictory informations. What is the purpose of each one ? What is the difference between rgw_frontends = ".. port = ..." and rgw_port = ? Or rgw_host and rgw_dns_name. What is the difference ? The documentation provides no help at all: rgw_dns_name Description: The DNS name of the served domain. See also the hostnames setting within regions. The description says nothing new, it just repeat the field name. Is one of them used by the manager for communication ? I already had the problem for the entry in the certificate used by the frontend, it used an IP coming from nowhere. If a fcgi is used, how the manager find the endpoint ?

2 2