- ceph-users - lists.ceph.io

Ceph server
by Ignazio Cassano 12 Mar '21

12 Mar '21

Hello, what do you think about of ceph cluster made up of 6 nodes each one with the following configuration ? A+ Server 1113S-WN10RT Barebone Supermicro A+ Server 1113S-WN10RT - 1U - 10x U.2 NVMe - 2x M.2 - Dual 10-Gigabit LAN - 750W Redundant Processor AMD EPYC™ 7272 Processor 12-core 2.90GHz 64MB Cache (120W) Memory 8 x 8GB PC4-25600 3200MHz DDR4 ECC RDIMM U.2/U.3 NVMe Drive 5 x 8.0TB Intel® SSD DC P4510 Series U.2 PCIe 3.1 x4 NVMe Solid State Drive Hard Drive 2 x 240GB Intel® SSD D3-S4610 Series 2.5" SATA 6.0Gb/s Solid State Drive Network Card 2 x Intel® 10-Gigabit Ethernet Converged Network Adapter X710-DA2 (2x SFP+) Server Management Supermicro Update Manager (SUM) (OOB Management Package), included Operating System No Windows Operating System Warranty 3 Year Depot Warranty (Return for Repair) RAID U.2 [NVMe, 10 ports] No RAID (*OS) -> 5 x 8.0TB Intel® SSD DC P4510 Series U.2 PCIe 3.1 x4 NVMe Solid State Drive Thanks Ignazio

4 9

ceph boostrap initialization :: nvme drives not empty after >12h
by Adrian Sevcenco 12 Mar '21

12 Mar '21

Hi! yesterday i bootstrapped (with cephadm) my first ceph installation and things looked somehow ok .. but today the osds are not yet ready and i have in dashboard this warnings: MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs PG_AVAILABILITY: Reduced data availability: 64 pgs inactive PG_DEGRADED: Degraded data redundancy: 2/14 objects degraded (14.286%), 66 pgs undersized TOO_FEW_OSDS: OSD count 2 < osd_pool_default_size 3 and in logs: 3/12/21 12:18:19 PM [INF] OSD <1> is not empty yet. Waiting a bit more 3/12/21 12:18:19 PM [INF] OSD <0> is not empty yet. Waiting a bit more 3/12/21 12:18:19 PM [INF] Can't even stop one OSD. Cluster is probably busy. Retrying later.. 3/12/21 12:18:19 PM [ERR] cmd: osd ok-to-stop failed with: 31 PGs are already too degraded, would become too degraded or might become unavailable. (errno:-16) this is a single node, whole package ceph install with 2 local nvme drives as osds (to be used 2x replicated like a raid1 array) So, can anyone tell me what is going on? Thanks a lot!! Adrian

3 4

Recover data from Cephfs snapshot
by Jesper Lykkegaard Karlsen 12 Mar '21

12 Mar '21

Hi Ceph'ers, I love the possibility to make snapshots on Cephfs systems. Although there is one thing that puzzles me. Creating snapshot takes no time to do and deleting snapshots can bring PGs into snaptrim state for some hours. While recovering data from a snapshot will always invoke a full data transfer, where data are "physically" being copied back into place. This can make recovering from snapshots on Cephfs a rather heavy procedure. I have even tried "mv" command but that also starts transfer real data instead of just moving metadata pointers. Am I missing some "ceph snapshot recover" command, that can move metadata pointers and make recovery much lighter, or is this just that way it is? Best reagards, Jesper -------------------------- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Gustav Wieds Vej 10 8000 Aarhus C E-mail: jelka(a)mbg.au.dk Tlf: +45 50906203

1 0

mds rank failed. loaded with preallocated inodes that are inconsistent with inotable
by Ch Wan 12 Mar '21

12 Mar '21

Hi cephers. Recently we are upgrading ceph cluster from mimic to nautilus. We have 5 ranks and decrease max_mds from 5 down to 1 smoothly. When we set max_mds from 2 to 1, the cluster show that rank 1 is failed, those are mds logs: 2021-03-12 16:21:26.974 7f366e949700 1 mds.1.125077 handle_mds_map state change up:boot --> up:replay 2021-03-12 16:21:26.974 7f366e949700 1 mds.1.125077 replay_start 2021-03-12 16:21:26.974 7f366e949700 1 mds.1.125077 recovery set is 0 2021-03-12 16:21:26.974 7f366e949700 1 mds.1.125077 waiting for osdmap 460461 (which blacklists prior instance) 2021-03-12 16:21:27.018 7f366893d700 0 mds.1.cache creating system inode with ino:0x101 2021-03-12 16:21:27.019 7f366893d700 0 mds.1.cache creating system inode with ino:0x1 2021-03-12 16:21:27.404 7f366713a700 0 mds.1.cache creating system inode with ino:0x100 2021-03-12 16:21:27.407 7f366713a700 -1 log_channel(cluster) log [ERR] : client client1: (2934972)loaded with preallocated inodes that are inconsistent with inotable 2021-03-12 16:21:27.407 7f366713a700 -1 log_channel(cluster) log [ERR] : client client2: (2862164)loaded with preallocated inodes that are inconsistent with inotable 2021-03-12 16:21:27.407 7f366713a700 -1 log_channel(cluster) log [ERR] : client client3: (2579839)loaded with preallocated inodes that are inconsistent with inotable 2021-03-12 16:21:27.407 7f366713a700 -1 log_channel(cluster) log [ERR] : client client4: (2579815)loaded with preallocated inodes that are inconsistent with inotable ... thanks for any help!

1 0

Re: how to tell balancer to balance
by Boris Behrens 12 Mar '21

12 Mar '21

Hi Joe, I've tested to create a plan on my own, but I still get the same message (Error EALREADY: Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect). What I've also tried to reweight 0.8 the three most filled OSDs, which worked good. After all misplaced objects were resolved I reweighted them back to 1, and now I am where I started. The point with the wasted disks is something I can not follow. If there are two types of disks, ceph balances pretty good between these disks. -6 58.89636 - 59 TiB 41 TiB 40 TiB 2.4 GiB 104 GiB 18 TiB 69.20 0.98 - host s3db5 0 hdd 3.73630 1.00000 3.7 TiB 2.2 TiB 2.1 TiB 70 MiB 6.3 GiB 1.5 TiB 60.06 0.85 46 up osd.0 25 hdd 3.73630 1.00000 3.7 TiB 2.2 TiB 2.1 TiB 4.1 MiB 6.1 GiB 1.5 TiB 58.94 0.84 39 up osd.25 26 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 179 MiB 7.3 GiB 1.0 TiB 72.32 1.03 47 up osd.26 27 hdd 3.73630 1.00000 3.7 TiB 2.1 TiB 2.0 TiB 6.8 MiB 6.1 GiB 1.6 TiB 56.17 0.80 47 up osd.27 28 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 935 MiB 28 GiB 3.9 TiB 73.68 1.05 197 up osd.28 29 hdd 14.65039 1.00000 15 TiB 9.9 TiB 9.8 TiB 515 MiB 24 GiB 4.7 TiB 67.83 0.96 188 up osd.29 30 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 774 MiB 26 GiB 3.9 TiB 73.56 1.04 196 up osd.30 -7 58.89636 - 59 TiB 43 TiB 42 TiB 15 GiB 120 GiB 16 TiB 72.73 1.03 - host s3db6 32 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 22 MiB 7.8 GiB 1.0 TiB 72.18 1.02 60 up osd.32 33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 381 MiB 8.1 GiB 670 GiB 82.50 1.17 57 up osd.33 34 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 444 MiB 8.5 GiB 604 GiB 84.21 1.20 60 up osd.34 35 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 296 MiB 8.6 GiB 513 GiB 86.59 1.23 53 up osd.35 36 hdd 14.65039 1.00000 15 TiB 9.9 TiB 9.8 TiB 541 MiB 24 GiB 4.8 TiB 67.55 0.96 209 up osd.36 37 hdd 14.65039 1.00000 15 TiB 10 TiB 10 TiB 12 GiB 36 GiB 4.5 TiB 69.22 0.98 191 up osd.37 38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 27 GiB 4.0 TiB 72.62 1.03 209 up osd.38 [root@s3db1 ~]# ceph balancer optimize boristest Error EINVAL: Balancer enabled, disable to optimize manually [root@s3db1 ~]# ceph balancer off [root@s3db1 ~]# ceph balancer optimize bbtest Error EALREADY: Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect [root@s3db1 ~]# ceph balancer on Am Do., 11. März 2021 um 21:59 Uhr schrieb Joe Comeau <Joe.Comeau(a)hli.ubc.ca >: > I read your email > > maybe DO NOT use the balancer > pools should be of different disks > > then you could balance within the pools > > disks should all be the same size > the 4TB will fill up - the 8TB will only go to 50% (or 4 TB) - so in > effect wasting 4TB of the 8 TB disk > > our cluster & our pool > All our disks no matter what are 8 TB in size. > > > > > > >>> Boris Behrens <bb(a)kervyn.de> 3/11/2021 5:53 AM >>> > Hi, > I know this topic seems to be handled a lot (as far as I can see), but I > reached the end of my google_foo. > > * We have OSDs that are near full, but there are also OSDs that are only > loaded with 50%. > * We have 4,8,16 TB rotating disks in the cluster. > * The disks that get packed are 4TB disks and very empty disks are also 4TB > * The OSD nodes are all around the same total disk space (51 - 59) > * The balancer tells me that it can not find further optimization, or that > pg_num is decreasin. > > How can I debug further before the cluster goes into a bad state? > > [root@s3db1 ~]# ceph osd df tree | sort -nk 17 | head -n 30 > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL > %USE VAR PGS STATUS TYPE NAME > MIN/MAX VAR: 0.75/1.23 STDDEV: 6.96 > TOTAL 673 TiB 474 TiB 452 TiB 100 GiB 1.2 TiB 199 > TiB 70.39 > -3 58.49872 - 58 TiB 39 TiB 36 TiB 8.2 GiB 85 GiB 19 > TiB 67.40 0.96 - host s3db2 > -4 58.49872 - 58 TiB 40 TiB 35 TiB 35 GiB 81 GiB 19 > TiB 67.91 0.96 - host s3db3 > -11 50.94173 - 51 TiB 35 TiB 35 TiB 3.5 GiB 94 GiB 16 > TiB 68.00 0.97 - host s3db10 > -10 51.28331 - 51 TiB 35 TiB 35 TiB 4.6 GiB 93 GiB 16 > TiB 69.16 0.98 - host s3db9 > -6 58.89636 - 59 TiB 41 TiB 40 TiB 2.4 GiB 102 GiB 18 > TiB 69.15 0.98 - host s3db5 > -12 50.99052 - 51 TiB 36 TiB 36 TiB 1.8 GiB 93 GiB 15 > TiB 69.99 0.99 - host s3db11 > -2 58.20561 - 58 TiB 41 TiB 37 TiB 9.6 GiB 96 GiB 17 > TiB 70.00 0.99 - host s3db1 > -1 673.44452 - 673 TiB 474 TiB 452 TiB 100 GiB 1.2 TiB 199 > TiB 70.39 1.00 - root default > -5 58.49872 - 58 TiB 42 TiB 35 TiB 7.0 GiB 94 GiB 17 > TiB 71.06 1.01 - host s3db4 > -8 58.89636 - 59 TiB 42 TiB 42 TiB 3.6 GiB 108 GiB 17 > TiB 71.91 1.02 - host s3db7 > -7 58.89636 - 59 TiB 43 TiB 42 TiB 15 GiB 120 GiB 16 > TiB 72.69 1.03 - host s3db6 > -37 58.55478 - 59 TiB 43 TiB 43 TiB 4.4 GiB 117 GiB 16 > TiB 73.18 1.04 - host s3db12 > -9 51.28331 - 51 TiB 38 TiB 38 TiB 4.9 GiB 103 GiB 13 > TiB 74.18 1.05 - host s3db8 > 15 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 2.1 GiB 0 B 1.7 > TiB 52.87 0.75 45 up osd.15 > 6 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 1.7 GiB 0 B 1.7 > TiB 52.90 0.75 46 up osd.6 > 12 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 570 MiB 0 B 1.7 > TiB 53.04 0.75 41 up osd.12 > 81 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 895 MiB 0 B 1.7 > TiB 54.26 0.77 51 up osd.81 > 27 hdd 3.73630 1.00000 3.7 TiB 2.1 TiB 2.0 TiB 6.8 MiB 5.8 GiB 1.6 > TiB 56.12 0.80 47 up osd.27 > 3 hdd 3.63689 1.00000 3.6 TiB 2.1 TiB 1.6 TiB 510 MiB 0 B 1.6 > TiB 57.04 0.81 51 up osd.3 > 5 hdd 3.63689 1.00000 3.6 TiB 2.1 TiB 1.5 TiB 431 MiB 0 B 1.5 > TiB 57.88 0.82 49 up osd.5 > 80 hdd 3.63689 1.00000 3.6 TiB 2.1 TiB 1.5 TiB 1.8 GiB 0 B 1.5 > TiB 58.31 0.83 51 up osd.80 > 25 hdd 3.73630 1.00000 3.7 TiB 2.2 TiB 2.1 TiB 4.1 MiB 6.1 GiB 1.5 > TiB 58.91 0.84 39 up osd.25 > 0 hdd 3.73630 1.00000 3.7 TiB 2.2 TiB 2.1 TiB 83 MiB 6.2 GiB 1.5 > TiB 60.03 0.85 46 up osd.0 > 79 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.4 TiB 1.8 GiB 0 B 1.4 > TiB 62.53 0.89 47 up osd.79 > 61 hdd 7.32619 1.00000 7.3 TiB 4.6 TiB 4.6 TiB 1.1 GiB 12 GiB 2.7 > TiB 62.80 0.89 101 up osd.61 > 67 hdd 7.27739 1.00000 7.3 TiB 4.6 TiB 4.6 TiB 557 MiB 13 GiB 2.7 > TiB 63.29 0.90 96 up osd.67 > 72 hdd 7.32619 1.00000 7.3 TiB 4.6 TiB 4.6 TiB 107 MiB 11 GiB 2.7 > TiB 63.36 0.90 87 up osd.72 > > [root@s3db1 ~]# ceph osd df tree | sort -nk 17 | tail > 51 hdd 7.27739 1.00000 7.3 TiB 5.6 TiB 5.5 TiB 724 MiB 14 GiB 1.7 > TiB 76.34 1.08 105 up osd.51 > 71 hdd 3.68750 1.00000 3.7 TiB 2.8 TiB 2.8 TiB 3.7 MiB 7.8 GiB 867 > GiB 77.04 1.09 47 up osd.71 > 82 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 839 GiB 628 MiB 0 B 839 > GiB 77.48 1.10 45 up osd.82 > 14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 777 GiB 18 GiB 0 B 777 > GiB 79.14 1.12 59 up osd.14 > 4 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 752 GiB 826 MiB 0 B 752 > GiB 79.80 1.13 53 up osd.4 > 75 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9 TiB 523 MiB 8.2 GiB 757 > GiB 79.95 1.14 53 up osd.75 > 76 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0 TiB 237 MiB 9.2 GiB 668 > GiB 82.30 1.17 50 up osd.76 > 33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 380 MiB 8.5 GiB 671 > GiB 82.46 1.17 57 up osd.33 > 34 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 464 MiB 8.4 GiB 605 > GiB 84.18 1.20 60 up osd.34 > 35 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 352 MiB 8.7 GiB 515 > GiB 86.55 1.23 53 up osd.35 > > [root@s3db1 ~]# ceph balancer status > { > "last_optimize_duration": "0:00:00.020142", > "plans": [], > "mode": "upmap", > "active": true, > "optimize_result": "Unable to find further optimization, or pool(s) > pg_num is decreasing, or distribution is already perfect", > "last_optimize_started": "Thu Mar 11 13:42:32 2021" > } > > [root@s3db1 ~]# ceph df > RAW STORAGE: > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 673 TiB 199 TiB 474 TiB 474 TiB 70.41 > TOTAL 673 TiB 199 TiB 474 TiB 474 TiB 70.41 > > POOLS: > POOL ID PGS STORED > OBJECTS USED %USED MAX AVAIL > rbd 0 64 0 B > 0 0 B 0 19 TiB > .rgw.root 1 64 98 KiB > 118 98 KiB 0 19 TiB > eu-central-1.rgw.control 2 64 0 B > 8 0 B 0 19 TiB > eu-central-1.rgw.data.root 3 64 1022 KiB > 3.02k 1022 KiB 0 19 TiB > eu-central-1.rgw.gc 4 64 84 MiB > 32 84 MiB 0 19 TiB > eu-central-1.rgw.log 5 64 220 MiB > 564 220 MiB 0 19 TiB > eu-central-1.rgw.users.uid 6 64 2.8 MiB > 6.89k 2.8 MiB 0 19 TiB > eu-central-1.rgw.users.keys 7 64 262 KiB > 6.70k 262 KiB 0 19 TiB > eu-central-1.rgw.meta 8 64 384 KiB > 1k 384 KiB 0 19 TiB > eu-central-1.rgw.users.email 9 64 40 B > 1 40 B 0 19 TiB > eu-central-1.rgw.buckets.index 10 64 10 GiB > 67.54k 10 GiB 0.02 19 TiB > eu-central-1.rgw.buckets.data 11 1024 151 TiB > 106.48M 151 TiB 72.67 19 TiB > eu-central-1.rgw.buckets.non-ec 12 64 268 MiB > 13.03k 268 MiB 0 19 TiB > eu-central-1.rgw.usage 13 64 501 MiB > 32 501 MiB 0 19 TiB > eu-msg-1.rgw.control 56 64 0 B > 8 0 B 0 19 TiB > eu-msg-1.rgw.data.root 57 64 71 KiB > 221 71 KiB 0 19 TiB > eu-msg-1.rgw.gc 58 64 60 KiB > 32 60 KiB 0 19 TiB > eu-msg-1.rgw.log 59 64 835 KiB > 242 835 KiB 0 19 TiB > eu-msg-1.rgw.users.uid 60 64 56 KiB > 107 56 KiB 0 19 TiB > eu-msg-1.rgw.usage 61 64 36 MiB > 25 36 MiB 0 19 TiB > eu-msg-1.rgw.users.keys 62 64 3.8 KiB > 97 3.8 KiB 0 19 TiB > eu-msg-1.rgw.meta 63 64 600 KiB > 1.58k 600 KiB 0 19 TiB > eu-msg-1.rgw.buckets.index 64 64 46 MiB > 112 46 MiB 0 19 TiB > eu-msg-1.rgw.users.email 65 64 0 B > 0 0 B 0 19 TiB > eu-msg-1.rgw.buckets.data 66 64 2.8 TiB > 1.14M 2.8 TiB 4.76 19 TiB > eu-msg-1.rgw.buckets.non-ec 67 64 2.2 MiB > 353 2.2 MiB 0 19 TiB > default.rgw.control 69 32 0 B > 8 0 B 0 19 TiB > default.rgw.data.root 70 32 0 B > 0 0 B 0 19 TiB > default.rgw.gc 71 32 0 B > 0 0 B 0 19 TiB > default.rgw.log 72 32 0 B > 0 0 B 0 19 TiB > default.rgw.users.uid 73 32 0 B > 0 0 B 0 19 TiB > fra-1.rgw.control 74 32 0 B > 8 0 B 0 19 TiB > fra-1.rgw.meta 75 32 0 B > 0 0 B 0 19 TiB > fra-1.rgw.log 76 32 50 B > 28 50 B 0 19 TiB > whitespace-again-2021-03-10 77 64 111 MiB > 363.94k 111 MiB 0 19 TiB > whitespace-again-2021-03-10_2 78 32 18 KiB > 59 18 KiB 0 19 TiB > whitespace-again-2021-03-10_3 79 32 11 KiB > 36 11 KiB 0 19 TiB > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groÃƒ¼en Saal. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

ERROR: S3 error: 403 (SignatureDoesNotMatch)
by Szabo, Istvan (Agoda) 12 Mar '21

12 Mar '21

Hi, I'm struggling with my old cluster cnamed address. The s3 and curl commands are working properly with the not cnamed address, but with the cnamed one, I got this in the ciwetweb log: 2021-03-12 10:24:18.812329 7f6b0c527700 1 ====== starting new request req=0x7f6b0c520f90 ===== 2021-03-12 10:24:18.812387 7f6b0c527700 2 req 10:0.000058::HEAD /::initializing for trans_id = tx00000000000000000000a-00604adee2-8e4fc3-default 2021-03-12 10:24:18.812412 7f6b0c527700 10 rgw api priority: s3=5 s3website=4 2021-03-12 10:24:18.812417 7f6b0c527700 10 host=cnamedhostname 2021-03-12 10:24:18.812484 7f6b0c527700 10 handler=25RGWHandler_REST_Bucket_S3 2021-03-12 10:24:18.812490 7f6b0c527700 2 req 10:0.000163:s3:HEAD /::getting op 3 2021-03-12 10:24:18.812499 7f6b0c527700 10 op=25RGWStatBucket_ObjStore_S3 2021-03-12 10:24:18.812503 7f6b0c527700 2 req 10:0.000176:s3:HEAD /:stat_bucket:verifying requester 2021-03-12 10:24:18.812541 7f6b0c527700 2 req 10:0.000214:s3:HEAD /:stat_bucket:normalizing buckets and tenants 2021-03-12 10:24:18.812548 7f6b0c527700 10 s->object=<NULL> s->bucket= cnamedhostname 2021-03-12 10:24:18.812556 7f6b0c527700 2 req 10:0.000229:s3:HEAD /:stat_bucket:init permissions 2021-03-12 10:24:18.812594 7f6b0c527700 10 cache get: name=default.rgw.meta+root+ cnamedhostname : type miss (requested=0x16, cached=0x0) 2021-03-12 10:24:18.813525 7f6b0c527700 10 cache put: name=default.rgw.meta+root+ cnamedhostname info.flags=0x0 2021-03-12 10:24:18.813554 7f6b0c527700 10 moving default.rgw.meta+root+ cnamedhostname to cache LRU end 2021-03-12 10:24:18.813664 7f6b0c527700 10 read_permissions on cnamedhostname [] ret=-2002 2021-03-12 10:24:18.813833 7f6b0c527700 2 req 10:0.001506:s3:HEAD /:stat_bucket:op status=0 2021-03-12 10:24:18.813848 7f6b0c527700 2 req 10:0.001520:s3:HEAD /:stat_bucket:http status=404 2021-03-12 10:24:18.813855 7f6b0c527700 1 ====== req done req=0x7f6b0c520f90 op status=0 http_status=404 ====== 2021-03-12 10:24:18.813962 7f6b0c527700 1 civetweb: 0x557d45468000: 10.118.199.248 - - [12/Mar/2021:10:24:18 +0700] "HEAD / HTTP/1.1" 404 0 - curl/7.29.0 And I got this on the s3cmd verbose output: DEBUG: s3cmd version 2.1.0 DEBUG: ConfigParser: Reading file '.s3cfg-testuser-http' DEBUG: ConfigParser: access_key->29...17_chars...J DEBUG: ConfigParser: secret_key->fK...37_chars...R DEBUG: ConfigParser: host_base->cnamedhostname:80 DEBUG: ConfigParser: host_bucket->cnamedhostname:80/%(bucket) DEBUG: ConfigParser: use_https->False DEBUG: ConfigParser: signature_v2->True DEBUG: Updating Config.Config cache_file -> DEBUG: Updating Config.Config follow_symlinks -> False DEBUG: Updating Config.Config verbosity -> 10 DEBUG: Unicodising 'ls' using UTF-8 DEBUG: Command: ls DEBUG: CreateRequest: resource[uri]=/ DEBUG: Using signature v2 DEBUG: SignHeaders: u'GET\n\n\n\nx-amz-date:Fri, 12 Mar 2021 03:31:39 +0000\n/' DEBUG: Processing request, please wait... DEBUG: get_hostname(None): cnamedhostname DEBUG: ConnMan.get(): creating new connection: http://cnamedhostname DEBUG: non-proxied HTTPConnection(cnamedhostname, None) DEBUG: format_uri(): / DEBUG: Sending request method_string='GET', uri=u'/', headers={'Authorization': u'AWS 293WEU2ADWGIUO4RN39J:Q7kh7kzWXWSqMvUqqWwLOY6QKUE=', 'x-amz-date': 'Fri, 12 Mar 2021 03:31:39 +0000'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (http://cnamedhostname#1) DEBUG: Response: {'data': '<?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx00000000000000000000b-00604ae09b-8e4fbd-default</RequestId><HostId>8e4fbd-default-default</HostId></Error>', 'headers': {'accept-ranges': 'bytes', 'content-length': '198', 'content-type': 'application/xml', 'date': 'Fri, 12 Mar 2021 03:31:39 GMT', 'x-amz-request-id': 'tx00000000000000000000b-00604ae09b-8e4fbd-default'}, 'reason': 'Forbidden', 'status': 403} DEBUG: S3Error: 403 (Forbidden) DEBUG: HttpHeader: date: Fri, 12 Mar 2021 03:31:39 GMT DEBUG: HttpHeader: content-length: 198 DEBUG: HttpHeader: x-amz-request-id: tx00000000000000000000b-00604ae09b-8e4fbd-default DEBUG: HttpHeader: content-type: application/xml DEBUG: HttpHeader: accept-ranges: bytes DEBUG: ErrorXML: Code: 'SignatureDoesNotMatch' DEBUG: ErrorXML: RequestId: 'tx00000000000000000000b-00604ae09b-8e4fbd-default' DEBUG: ErrorXML: HostId: '8e4fbd-default-default' ERROR: S3 error: 403 (SignatureDoesNotMatch) Any idea? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 2

v14.2.17 Nautilus released
by David Galloway 11 Mar '21

11 Mar '21

Body: We're happy to announce the 17th backport release in the Nautilus series. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v14-2-17-nautilus-released Notable Changes --------------- * $pid expansion in config paths like `admin_socket` will now properly expand to the daemon pid for commands like `ceph-mds` or `ceph-osd`. Previously only `ceph-fuse`/`rbd-nbd` expanded `$pid` with the actual daemon pid. * RADOS: PG removal has been optimized in this release. * RADOS: Memory allocations are tracked in finer detail in BlueStore and displayed as a part of the ``dump_mempools`` command. * cephfs: clients which acquire capabilities too quickly are throttled to prevent instability. See new config option ``mds_session_cap_acquisition_throttle`` to control this behavior. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.17.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 2e95b5d99e0dec516803c8a1b57fbd2c8f45fd63

1 0

Failure Domain = NVMe?
by Dave Hall 11 Mar '21

11 Mar '21

Hello, In some documentation I was reading last night about laying out OSDs, it was suggested that if more that one OSD uses the same NVMe drive, the failure-domain should probably be set to node. However, for a small cluster the inclination is to use EC-pools and failure-domain = OSD. I was wondering if there is a middle ground - could we define failure-domain = NVMe? I think the map would need to be defined manually in the same way that failure-domain = rack requires information about which nodes are in each rack. Example: My latest OSD nodes have 8 HDDs and 3 U.2 NVMe. I'd set up the WAL/DB for with HDDs per OSD (wasted space on the 3rd NVMe). Across all my OSD nodes I will have 8 HDDs and either 2 or 3 NVMe devices per node - 15 total NVMe devices. My preferred EC-pool profile is 8+2. It seems that this profile could be safely dispersed across 15 failure domains, resulting in protection against NVMe failure. Please let me know if this is worth pursuing. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office)

5 9

mon db growing. over 500Gb
by ricardo.re.azevedo＠gmail.com 11 Mar '21

11 Mar '21

Hi all, I have a fairly pressing issue. I had a monitor fall out of quorum because it ran out of disk space during rebalancing from switching to upmap. I noticed all my monitor store.db started taking up nearly all disk space so I set noout, nobackfill and norecover and shutdown all the monitor daemons. Each store.db was at: mon.a 89GB (the one that firt dropped out) mon.a 400GB mon.c 400GB I tried setting mon_compact_on_start. This brought mon.a down to 1GB. Cool. However, when I try it on the other monitors it increased the db size ~1Gb/10s so I shut them down again. Any idea what is going on? Or how can I shrik back down the db?

4 6

Can FS snapshots cause factor 3 performance loss?
by Frank Schilder 11 Mar '21

11 Mar '21

Hi all, we are observing a dramatic performance drop on our ceph file system and are wondering if this could be related to ceph fs snapshots. We are taking rotating snapshots in 2 directories and have 11 snapshots in each (ls below) as of today. We observe the performance drop with an rsync process that writes to ceph fs to another folder *without* snapshots. The performance reduction is a factor of 3 or even higher. Could this possibly be caused by the snapshots being present? Has anyone else seen something like this? The reason we consider snapshots is that not much else changed on the cluster except that we started taking rolling snapshots on the 23rd of February. In addition, the kernel symbols ceph_update_snap_trace, rebuild_snap_realms and build_snap_context show up really high in a perf report. The performance reduction seems to be present since at least 3 days. The ceph version is mimic 13.2.10. The kernel version of the rsync server is 3.10.0-1127.10.1.el7.x86_64. $ ls home/.snap 2021-02-23_183554+0100_weekly 2021-03-06_000611+0100_daily 2021-03-09_000611+0100_daily 2021-03-01_000911+0100_weekly 2021-03-07_000611+0100_daily 2021-03-10_000611+0100_daily 2021-03-04_000611+0100_daily 2021-03-08_000611+0100_daily 2021-03-11_000611+0100_daily 2021-03-05_000611+0100_daily 2021-03-08_000911+0100_weekly $ ls groups/.snap 2021-02-23_183554+0100_weekly 2021-03-06_000611+0100_daily 2021-03-09_000611+0100_daily 2021-03-01_000912+0100_weekly 2021-03-07_000611+0100_daily 2021-03-10_000612+0100_daily 2021-03-04_000611+0100_daily 2021-03-08_000611+0100_daily 2021-03-11_000612+0100_daily 2021-03-05_000611+0100_daily 2021-03-08_000911+0100_weekly Many thanks for any pointers and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0