- ceph-users - lists.ceph.io

Balancer is active, but not balancing
by Thomas 03 Nov '19

03 Nov '19

Hi, I activated balancer in order to balance data distribution: root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } However, the data stored on 1.6TB HDD in specific pool "hdb_backup" is not balanced; the range starts with osd.265 size: 1.6 usage: 52.83 reweight: 1.00000 and ends with osd.145 size: 1.6 usage: 80.19 reweight: 1.00000 The affected drives are located on 4 nodes. The result is that not all available disk space is available for usage. I have attached pastebin <https://pastebin.com/dNyEwNR0> with - ceph osd df sorted by usage - ceph osd df tree Please advise how to start balancer to correct data distribution. THX

1 0

iSCSI write performance
by Ryan 02 Nov '19

02 Nov '19

I'm in the process of testing the iscsi target feature of ceph. The cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5 hosts with 12 SSD OSDs per host. Some basic testing moving VMs to a ceph backed datastore is only showing 60MB/s transfers. However moving these back off the datastore is fast at 200-300MB/s. What should I be looking at to track down the write performance issue? In comparison with the Nimble Storage arrays I can see 200-300MB/s in both directions. Thanks, Ryan

8 20

Weird blocked OP issue.
by Robert LeBlanc 02 Nov '19

02 Nov '19

We had an OSD host with 13 OSDs fail today and we have a weird blocked OP message that I can't understand. There are no OSDs with blocked ops, just `mon` (multiple times), and some of the rgw instances. cluster: id: 570bcdbb-9fdf-406f-9079-b0181025f8d0 health: HEALTH_WARN 1 large omap objects Degraded data redundancy: 2083023/195702437 objects degraded (1.064%), 880 pgs degraded, 880 pgs undersized 1609 pgs not deep-scrubbed in time 4 slow ops, oldest one blocked for 506699 sec, daemons [mon,sun-gcs02-rgw01,mon,sun-gcs02-rgw02,mon,sun-gcs02-rgw03] have slow ops. services: mon: 3 daemons, quorum sun-gcs02-rgw01,sun-gcs02-rgw02,sun-gcs02-rgw03 (age 6m) mgr: sun-gcs02-rgw02(active, since 5d), standbys: sun-gcs02-rgw03, sun-gcs02-rgw04 osd: 767 osds: 754 up (since 10m), 754 in (since 104m); 880 remapped pgs rgw: 16 daemons active (sun-gcs02-rgw01.rgw0, sun-gcs02-rgw01.rgw1, sun-gcs02-rgw01.rgw2, sun-gcs02-rgw01.rgw3, sun-gcs02-rgw02.rgw0, sun-gcs02-rgw02.rgw1, sun-gcs02-rgw02.rgw2, sun-gcs02-rgw02.rgw3, sun-gcs02-rgw03.rgw0, sun-gcs02-rgw03.rgw1, sun-gcs02-rgw03.rgw2, s un-gcs02-rgw03.rgw3, sun-gcs02-rgw04.rgw0, sun-gcs02-rgw04.rgw1, sun-gcs02-rgw04.rgw2, sun-gcs02-rgw04.rgw3) data: pools: 7 pools, 8240 pgs objects: 19.57M objects, 52 TiB usage: 88 TiB used, 6.1 PiB / 6.2 PiB avail pgs: 2083023/195702437 objects degraded (1.064%) 43492/195702437 objects misplaced (0.022%) 7360 active+clean 868 active+undersized+degraded+remapped+backfill_wait 12 active+undersized+degraded+remapped+backfilling io: client: 150 MiB/s rd, 642 op/s rd, 0 op/s wr recovery: 626 MiB/s, 223 objects/s $ ceph versions { "mon": { "ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)": 754 }, "mds": {}, "rgw": { "ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)": 16 }, "overall": { "ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)": 754, "ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)": 22 } } I restarted one of the monitors and it dropped out of the list only showing 2 blocked ops, but then showed up again a little while later. Any ideas on where to look? Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

1 1

Ceph Health error right after starting balancer
by Thomas Schneider 01 Nov '19

01 Nov '19

Hi, after enabling ceph balancer (with command ceph balancer on) the health status changed to error. This is the current output of ceph health detail: root@ld3955:~# ceph health detail HEALTH_ERR 1438 slow requests are blocked > 32 sec; 861 stuck requests are blocked > 4096 sec; mon ld5505 is low on available space REQUEST_SLOW 1438 slow requests are blocked > 32 sec 683 ops are blocked > 2097.15 sec 436 ops are blocked > 1048.58 sec 191 ops are blocked > 524.288 sec 78 ops are blocked > 262.144 sec 35 ops are blocked > 131.072 sec 11 ops are blocked > 65.536 sec 4 ops are blocked > 32.768 sec osd.62 has blocked requests > 65.536 sec osds 39,72 have blocked requests > 262.144 sec osds 6,19,67,173,174,187,188,269,434 have blocked requests > 524.288 sec osds 8,16,35,36,37,61,63,64,68,73,75,178,186,271,369,420,429,431,433,436 have blocked requests > 1048.58 sec osds 3,5,7,24,34,38,40,41,59,66,69,74,180,270,370,421,432,435 have blocked requests > 2097.15 sec REQUEST_STUCK 861 stuck requests are blocked > 4096 sec 25 ops are blocked > 8388.61 sec 836 ops are blocked > 4194.3 sec osds 2,28,29,32,60,65,181,185,268,368,423,424,426 have stuck requests > 4194.3 sec osds 0,30,70,71,184 have stuck requests > 8388.61 sec I understand that when balancer starts shifting PGs to other OSDs that this caused IO load on the cluster. However I don't understand why this is affecting OSD so heavily. And I don't understand why OSD of specific type (SSD, NVME) suffer although there's no balancing occuring on them. Regards Thomas

3 4

subtrees have overcommitted (target_size_bytes / target_size_ratio)
by Lars Täuber 01 Nov '19

01 Nov '19

Hello everybody! What does this mean? health: HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio and what does it have to do with the autoscaler? When I deactivate the autoscaler the warning goes away. $ ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 15106M 3.0 2454G 0.0180 0.3000 4.0 256 on cephfs_data 113.6T 1.5 165.4T 1.0306 0.9000 1.0 512 on $ ceph health detail HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_bytes Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_bytes 0 on pools [] POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_ratio Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_ratio 0.900 on pools ['cephfs_data'] Thanks Lars

3 10

librados aysnc I/O takes considerably longer to complete
by Ponnuvel Palaniyappan 01 Nov '19

01 Nov '19

Hi, Is anyone using librados AIO APIs? I seem to have a problem with that where the rados_aio_wait_for_complete() call just waits for a long period of time before it finishes without error. More info on my setup: I am using Ceph 14.2.4 and write 8MB objects. I run my AIO program on 24 nodes at the same time each writing a different data (splits into 8MB objects and ), each data is about 2G. Normally, it takes about 10 mins for all of them to complete. But often one or more nodes takes considerably longer to finish. When looking at the one of those, I mostly see that the IO requests have been submitted and waits at: #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00002aaaaad0c8fa in rados_aio_wait_for_complete () from /cgv/geovation/2/test/ceph/lib/librados.so.2 Then it eventually completes with no errors from rados_aio_wait_for_complete() call. The (pseudo) code looks like: while (data remains to be written) { size_t aio_ops_count = 0; rados_completion_t aio_comp[12]; for (size_t j = 0; j < 12; ++j) { int err = rados_aio_create_completion(NULL, NULL, NULL, &aio_comp[j]); if (err < 0) { cerr << "rados_aio_create_completion: " << strerror(-err) << endl; return 1; } string obj_ = getobjectid(); err = rados_aio_write_full(io, obj_.c_str(), aio_comp[j], read_buf[j], bytes); if (err < 0) { cerr << "rados_write_full: " << strerror(-err) << endl; return 1; } ++aio_ops_count; } for (size_t j = 0; j < aio_ops_count; ++j) { rados_aio_wait_for_complete(aio_comp[j]); int err = rados_aio_get_return_value(aio_comp[j]); // Considerably longer delay here ?? if (err < 0) { cerr << "rados_aio_get_return_value: " << strerror(-err) << endl; return 1; } rados_aio_release(aio_comp[j]); } } I ran under Valgrind and see no issues and also read the data back and checksum it to verify no corruption issues. So everything appears to "work" as expected except for longer delays at times. Wondering if anyone is using the AIO APIs to write objects and had experienced any similar problems. Please let me know if you need further information. (Originally posted this to dev(a)ceph.io and on Daniel's suggestion, I am posting here). Regards, Ponnuvel P -- Regards, Ponnuvel P

1 0

V/v Multiple pool for data in Ceph object
by tuan dung 01 Nov '19

01 Nov '19

hi ceph-users, i have a cluster run ceph object using version 14.2.1. I want to creat 2 pool for bucket data for purposes for security: + one bucket-data pool for public client access from internet (name *zone1.rgw.buckets.data-pub) * + one bucket-data pool for private client access from local network (name *zone1.rgw.buckets.data-pub)* each pool bucket-data has one individual access key: access key public (access pool public) and access key private (access pool private). Can you give me a recomment for this or bestpractice that you've done? what needs to be done? Or give me your best solution for securiy a cluster ceph object with public client access and private client access? Thank you very much Br, ---------------------------------------------- Dương Tuấn Dũng Email: dungdt.aicgroup(a)gmail.com ĐT: 0986153686

2 2

ceph pg dump hangs on mons w/o mgr
by Frank R 31 Oct '19

31 Oct '19

I have a 12.2.12 cluster with 3 mons where mgr will be active on 1. I have noticed that the command "ceph pg dump" hangs on all mons except the one where the mgr is running. "ceph pg dump" also runs fine on osd nodes. Is this expected behavior? thx Frank

1 0

Error in MGR Log: auth: could not find secret_id=<number>
by Thomas Schneider 31 Oct '19

31 Oct '19

Hello, I noticed several error messages in MGR (active) log: 2019-10-31 10:40:16.341 7ff9edd64700 0 auth: could not find secret_id=3714 2019-10-31 10:40:16.341 7ff9edd64700 0 cephx: verify_authorizer could not get service secret for service mgr secret_id=3714 2019-10-31 10:40:16.541 7ff9ecd62700 0 auth: could not find secret_id=3714 2019-10-31 10:40:16.541 7ff9ecd62700 0 cephx: verify_authorizer could not get service secret for service mgr secret_id=3714 2019-10-31 10:40:16.941 7ff9ecd62700 0 auth: could not find secret_id=3714 2019-10-31 10:40:16.941 7ff9ecd62700 0 cephx: verify_authorizer could not get service secret for service mgr secret_id=3714 2019-10-31 10:40:17.741 7ff9ecd62700 0 auth: could not find secret_id=3714 2019-10-31 10:40:17.741 7ff9ecd62700 0 cephx: verify_authorizer could not get service secret for service mgr secret_id=3714 These error message are logged every 5 sec. Can you please advise how to troubleshoot this issue? THX

1 0

Splitting PGs not happening on Nautilus 14.2.2
by Bryan Stillwell 30 Oct '19

30 Oct '19

This morning I noticed that on a new cluster the number of PGs for the default.rgw.buckets.data pool was way too small (just 8 PGs), but when I try to split the PGs the cluster doesn't do anything: # ceph osd pool set default.rgw.buckets.data pg_num 16 set pool 13 pg_num to 16 It seems to set the pg_num_target/pgp_num_target to 16, but pg_num/pgp_num never increase: # ceph osd dump | grep default.rgw.buckets.data pool 13 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 pg_num_target 16 pgp_num_target 16 autoscale_mode warn last_change 43217 flags hashpspool,creating stripe_width 0 compression_mode aggressive application rgw The clusters has 295 OSDs, so I need to do a bit of splitting today. Any ideas why the splits aren't starting? Thanks, Bryan

1 1