- ceph-users - lists.ceph.io

RGW: user modify default_storage_class does not work
by Huy Nguyen 15 Nov '23

15 Nov '23

Hi community, I'm using Ceph version 16.2.13. I tried to set default_storage_class but seems like it didn't work. Here is steps I did: I already had a storage class name COLD, then I modify the user default_storage_class like this: radosgw-admin user modify --uid testuser --placement-id default-placement --storage-class COLD after that, user info has show correctly: radosgw-admin user info --uid testuser { ... "op_mask": "read, write, delete", "default_placement": "default-placement", "default_storage_class": "COLD", ... Then I put a file using boto3, without specify any storage class: s3.Object(bucket_name, 'testdefault-object').put(Body="0"*1000)) But the object still jump into the STANDARD storage class. I don't know if this is a bug or did I miss something? Thanks

2 2

Stretch mode size
by Eugen Block 15 Nov '23

15 Nov '23

Hi, I'd like to ask for confirmation how I understand the docs on stretch mode [1]. It requires exact size 4 for the rule? Other sizes are not supported/won't work, for example size 6? Are there clusters out there which use this stretch mode? Once stretch mode is enabled, it's not possible to get out of it. How would one deal with a burnt down datacenter which can take months to rebuild? In a "self-managed" stretch cluster (let's say size 6) I could simply change the crush rule to not consider the failed datacenter anymore, deploy an additional mon somewhere and maybe reduce the size/min_size. Am I missing something? Thanks, Eugen [1] https://docs.ceph.com/en/reef/rados/operations/stretch-mode/#id2

2 4

Ceph Allocation - used space is unreasonably higher than stored space
by Motahare S 14 Nov '23

14 Nov '23

Hello everyone, Recently we have noticed that the results of "ceph df" stored and used space does not match; as the amount of stored data *1.5 (ec factor) is still like 5TB away from used amount: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL default.rgw.buckets.data 12 1024 144 TiB 70.60M 221 TiB 18.68 643 TiB blob and alloc configs are as below: bluestore_min_alloc_size_hdd : 65536 bluestore_min_alloc_size_ssd : 4096 luestore_max_blob_size_hdd : 524288 bluestore_max_blob_size_ssd : 65536 bluefs_shared_alloc_size : 65536 From sources across web about how ceph actually writes on the disk, I presumed that It will zero-pad the extents of an object to match the 4KB bdev_block_size, and then writes it in a blob which matches the min_alloc_size, however it can re-use parts of the blob's unwritten (but allocated because of min_alloc_size) space for another extent later. The problem though, was that we tested different configs in a minimal ceph octopus cluster with a 2G osd and bluestore_min_alloc_size_hdd = 65536. When we uploaded a 1KB file with aws s3 client, the amount of used/stored space was 64KB/1KB. We then uploaded another 1KB, and it went 128K/2K; kept doing it until 100% of the pool was used, but only 32MB stored. I expected ceph to start writing new 1KB files in the wasted 63KB(60KB)s of min_alloc_size blocks, but the cluster was totally acting as a full cluster and could no longer receive any new object. Is this behaviour expected for s3? Does ceph really use 64x space if your dataset is made of 1KB files? and all your object sizes should be a multiple of 64KB? Note that 5TB / (70.6M*1.5) ~ 50 so for every rados object about 50KB is wasted on average. we didn't observe this problem in RBD pools, probably because it cuts all objects in 4MB. I know that min_alloc_hdd is changed to 4KB in pacific, but I'm still curious how allocation really works and why it doesn't behave as expected? Also, re-deploying OSDs is a headache. Sincerely Motahare

3 3

CephFS mirror very slow (maybe for small files?)
by Stuart Cornell 14 Nov '23

14 Nov '23

Hi all. I have successfully configured an operational mirror between 2 sites for Ceph FS. The mirroring is running but the speed of data transfer is varying a lot over time (200KB/s – 120MB/s). The network infrastructure between the two Ceph clusters is reliable and should not be the cause of this speed variation. The FS in question, has a lot of small files in it and I suspect this is the cause of the variability – ie, the transfer of many small files will be more impacted by greater site-site latency. If this suspicion is true, what options do I have to improve the overall throughput? * Is it possible to parallelise or “chunk” the transfers with some options to the mirror daemon? * Would the use of multiple snapshot mirror points help? * Note that I am currently forced to use only a single point ( ceph fs snapshot mirror add <fs_name> <path> ) because the part of the FS in use is managed by Openstack Manila which created a subvolume. Requests to add mirrors for sub-directories are therefore denied. Any suggestions for how I can improve this throughput would be most welcome. Currrently I am running Pacific (16.2.10) on the sender and Quincy (17.2.6) on the target. Stuart Cornell Cloud Development Director http://graphcore.ai ** We have updated our privacy policy, which contains important information about how we collect and process your personal data. To read the policy, please click here<http://www.graphcore.ai/privacy> ** This email and its attachments are intended solely for the addressed recipients and may contain confidential or legally privileged information. If you are not the intended recipient you must not copy, distribute or disseminate this email in any way; to do so may be unlawful. Any personal data/special category personal data herein are processed in accordance with UK data protection legislation. All associated feasible security measures are in place. Further details are available from the Privacy Notice on the website and/or from the Company. Graphcore Limited (registered in England and Wales with registration number 10185006) is registered at 107 Cheapside, London, UK, EC2V 6DN. This message was scanned for viruses upon transmission. However Graphcore accepts no liability for any such transmission.

3 4

reduce mds_beacon_interval and mds_beacon_grace
by Dmitry Melekhov 14 Nov '23

14 Nov '23

Hello! I guess that default 4 and 15 seconds are too long for my configuration. If I want to reduce mds_beacon_grace to 5 seconds is 1 right value for mds_beacon_interval ? Thank you!

1 0

shrink db size
by Curt 13 Nov '23

13 Nov '23

Hello, As far as I can tell there is no way to shrink a db/wal after creation. I recently added a new server to my cluster with SSD's for the wal/db and just used the ceph dashboard for deployment. I did not specify a db size, which is my mistake, it seems by default it uses "block.db has no size configuration, will fallback to using as much as possible". So now my issue is I added 2 more drives, but with no space left on the SSD's get "2 fast devices were passed, but none are available". I had 7 4TB HDD and 2 2Tb SSD's and added 2 more 4TB. Distribution is 4 and 3 between the 2 SSD's. I just want to confirm, the best option is to set the bluestore_block_db_size/wal_size, zap and let them be recreated with a size of 300 and 2. I chose those just because I have space. I'm not going to do them all at the same time. Cheers, Curt

1 0

CEPH Cluster mon is out of quorum
by Mosharaf Hossain 13 Nov '23

13 Nov '23

Dear Concern I am observing a mon is out of quorum. The current running version of Ceph is octopus. Total Node in the cluster: 13 Mon: 3/3 Network: 20G(10Gx 2) bonded link Each node capacity: 512GB RAM + 72 Core CPU root@ceph6:/var/run/ceph# systemctl status ceph-mon(a)ceph6.service â— ceph-mon(a)ceph6.service - Ceph cluster monitor daemon Loaded: loaded (/lib/systemd/system/ceph-mon@.service; indirect; vendor preset: enabled) Active: active (running) since Mon 2023-11-13 14:18:28 +06; 55min ago Main PID: 165299 (ceph-mon) Tasks: 26 CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon(a)ceph6.service â””â”€165299 /usr/bin/ceph-mon -f --cluster ceph --id ceph6 --setuser ceph --setgroup ceph Nov 13 15:13:09 ceph6.bol-online.com ceph-mon[165299]: 2023-11-13T15:13:09.190+0600 7f7855600700 -1 mon.ceph6@2(probing) e6 get_health_metrics reporting 8 slow ops, oldest is log(1 entries from seq 1 at 2023-11-13T14:27:42.428071+0600) Nov 13 15:13:14 ceph6.bol-online.com ceph-mon[165299]: 2023-11-13T15:13:14.190+0600 7f7855600700 -1 mon.ceph6@2(probing) e6 get_health_metrics reporting 8 slow ops, oldest is log(1 entries from seq 1 at 2023-11-13T14:27:42.428071+0600) Nov 13 15:13:34 ceph6.bol-online.com ceph-mon[165299]: 2023-11-13T15:13:34.190+0600 7f7855600700 -1 mon.ceph6@2(probing) e6 get_health_metrics reporting 8 slow ops, oldest is log(1 entries from seq 1 at 2023-11-13T14:27:42.428071+0600) Nov 13 15:13:39 ceph6.bol-online.com ceph-mon[165299]: 2023-11-13T15:13:39.191+0600 7 root@ceph6:/var/run/ceph# ceph -s cluster: id: f8096ec7-51db-4557-85e6-57d7fdfe9423 health: HEALTH_WARN 1/3 mons down, quorum ceph2,mon1 nodeep-scrub flag(s) set 614 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum ceph2,mon1 (age 8h), out of quorum: ceph6 mgr: ceph4(active, since 2w), standbys: mon1, ceph3, ceph1, ceph6 mds: cephfs:1 {0=ceph8=up:active} 1 up:standby osd: 107 osds: 102 up (since 3h), 102 in (since 3h); 24 remapped pgs flags nodeep-scrub rgw: 4 daemons active (ceph10.rgw0, ceph7.rgw0, ceph9.rgw0, mon1.rgw0) rgw-nfs: 2 daemons active (ceph7, ceph9) root@ceph6:/var/run/ceph# ceph daemon mon.ceph6 mon_status { "name": "ceph6", "rank": 2, "state": "probing", "election_epoch": 0, "quorum": [], "features": { "required_con": "2449958747315978244", "required_mon": [ "kraken", "luminous", Total 3 mon in the cluster and all addresses are able to ping and telnet between them. 1. mon1: 10.10.10.71 2. ceph2: 10.10.10.52 3. ceph6: 10.10.10.56 Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 Cell: +8801787680828, Email: mosharaf.hossain(a)bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts…>

2 1

OSD disk is active in node but ceph show osd down and out
by Mosharaf Hossain 13 Nov '23

13 Nov '23

Hello In my CEPH Nautilus cluster, some OSDs have unexpectedly gone offline, despite all SSD disks appearing to be active on the node. What might be the cause of this issue? #ceph osd tree [showing disk for ceph6 status] -11 19.20854 host ceph6 10 ssd 1.74609 osd.10 down 0 1.00000 35 ssd 1.74609 osd.35 up 1.00000 1.00000 40 ssd 1.74609 osd.40 down 0 1.00000 45 ssd 1.74609 osd.45 down 0 1.00000 50 ssd 1.74609 osd.50 down 0 1.00000 54 ssd 1.74609 osd.54 up 1.00000 1.00000 58 ssd 1.74609 osd.58 up 1.00000 1.00000 59 ssd 1.74609 osd.59 up 1.00000 1.00000 67 ssd 1.74660 osd.67 up 1.00000 1.00000 68 ssd 1.74660 osd.68 down 0 1.00000 90 ssd 1.74660 osd.90 up 1.00000 1.00000 root@ceph6:~# fdisk -l | grep 1920383410176 Disk /dev/sdb: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sda: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdd: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdc: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdf: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdg: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdh: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdi: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sde: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdj: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdk: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Disk /dev/sdl: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 Cell: +8801787680828, Email: mosharaf.hossain(a)bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts…>

2 4

CEPH Cluster performance review
by Mosharaf Hossain 13 Nov '23

13 Nov '23

Hello Community Currently, I operate a CEPH Cluster utilizing Ceph Octopus version 1.5.2.7, installed through Ansible. The challenge I'm encountering is that, during scrubbing, OSD latency spikes to 300-600 ms, resulting in sluggish performance for all VMs. Additionally, some OSDs fail during the scrubbing process. In such instances, promptly halting the scrubbing resolves the issue. *Summary of CEPH Version*: CEPH Version Number of Nodes Node Networking OSD OSD Total pools PG Size 15.2.7 12 (6 SSD node + 6 HDD node) All nodes are connected through 10G bonded link, i.e. 10Gx2=20GB for each node. 64 SSD 42 HDD 106 one-ssd 256 active+clean one-hdd 512 active+clean cloudstack.hdd 512 active+clean I intend to enlarge the PG size for the "one-ssd" configuration. Please provide the PG number, and suggest the optimal approach to increase the PG size without causing any slowdown or service disruptions to the VMs. Your expertise and guidance on this matter would be highly valuable, and I'm eager to benefit from the collective knowledge of the Ceph community. Thank you in advance for your time and assistance. I look forward to hearing from you. *CEPH Health status:* root@mon1:~# ceph -s cluster: id: f8096ec7-51db-4557-85e6-57d7fdfe9423 health: HEALTH_WARN nodeep-scrub flag(s) set 656 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum ceph2,mon1,ceph6 (age 4d) mgr: ceph4(active, since 2w), standbys: mon1, ceph3, ceph6, ceph1 mds: cephfs:1 {0=ceph8=up:active} 1 up:standby osd: 107 osds: 105 up (since 3h), 105 in (since 3d) flags nodeep-scrub rgw: 4 daemons active (ceph10.rgw0, ceph7.rgw0, ceph9.rgw0, mon1.rgw0) rgw-nfs: 2 daemons active (ceph7, ceph9) task status: data: pools: 13 pools, 2057 pgs objects: 9.40M objects, 35 TiB usage: 106 TiB used, 154 TiB / 259 TiB avail pgs: 2057 active+clean io: client: 14 MiB/s rd, 30 MiB/s wr, 1.50k op/s rd, 1.53k op/s wr root@ceph1:~# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 151 TiB 78 TiB 72 TiB 73 TiB 48.04 ssd 110 TiB 78 TiB 32 TiB 32 TiB 29.42 TOTAL 261 TiB 156 TiB 104 TiB 105 TiB 40.19 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 64 3.8 KiB 0 11 KiB 0 23 TiB cephfs_metadata 2 8 228 MiB 79 685 MiB 0 23 TiB .rgw.root 3 32 6.0 KiB 8 1.5 MiB 0 23 TiB default.rgw.control 4 32 0 B 8 0 B 0 23 TiB default.rgw.meta 5 32 12 KiB 48 7.5 MiB 0 23 TiB default.rgw.log 6 32 4.8 KiB 207 6.0 MiB 0 23 TiB default.rgw.buckets.index 7 32 410 MiB 15 1.2 GiB 0 23 TiB default.rgw.buckets.data 8 512 4.6 TiB 1.29M 14 TiB 16.59 23 TiB default.rgw.buckets.non-ec 9 32 1.0 MiB 676 130 MiB 0 23 TiB one-hdd 10 512 9.2 TiB 2.45M 28 TiB 28.69 23 TiB device_health_metrics 11 1 9.5 MiB 113 28 MiB 0 23 TiB one-ssd 12 256 11 TiB 2.88M 32 TiB 31.37 23 TiB cloudstack.hdd 15 512 10 TiB 2.72M 31 TiB 30.94 23 TiB Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 Cell: +8801787680828, Email: mosharaf.hossain(a)bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts…>

4 4

Automatic triggering of the Ubuntu SRU process, e.g. for the recent 17.2.7 Quincy point release?
by Christian Rohmann 12 Nov '23

12 Nov '23

Hey Yuri, hey ceph-users, first of all, thanks for all your work on developing and maintaining Ceph. I was just wondering if there was any sort of process or trigger to the Ubuntu release team following a point release, for them to also create updated packages. If you look at https://packages.ubuntu.com/jammy-updates/ceph, there still only is 17.2.6 as the current update available. There was an [SRU] bug raised for 17.2.6 (https://bugs.launchpad.net/cloud-archive/+bug/2018929) I now opened a similar one (https://bugs.launchpad.net/cloud-archive/+bug/2043336) hoping I went the right way of triggering the packaging this point release. Even though the Ceph team does not build Quincy packages for Ubuntu 22.04 LTS (Jammy) themselves, it would be nice to still treat it somewhat of as a release channel and to automatically trigger these kind of processes. Regards Christian

1 0