June 2023 - ceph-users - lists.ceph.io

RadosGW strange behavior when using a presigned url generated by SDK PHP

by Huy Nguyen

Hi, I tried to generate a presigned url using SDK PHP, but it doesn't work. (I also tried to use boto3 with the same configures and the url works normally) Here is my php code: <?php require 'aws-autoloader.php'; use Aws\S3\S3Client; use Aws\Exception\AwsException; $s3Client = new Aws\S3\S3Client([ 'version' => '2006-03-01', 'region' => 'us-east-1', 'signature_version' => 'v4', 'use_path_style_endpoint' => true, 'endpoint' => 'http://hn.ss.bfcplatform.vn', 'credentials' => [ 'key' => 'DNMZAFE6G2PP8H9P05UU', 'secret' => 'XXX', ] ]); $cmd = $s3Client->getCommand('PutObject', [ 'Bucket' => 'huynnp-testbucket1', 'Key' => 'testfile.txt', ]); $request = $s3Client->createPresignedRequest($cmd, '+60 minutes'); // Set the expiration time as desired $presignedUrl = (string)$request->getUri(); echo "$presignedUrl"; ?> and then: curl -X PUT -T testfile.txt `php s3.php` <?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><RequestId>tx00000b7bb3b2deb6a6ef2-00649d5ebd-d1d50041-hn-1</RequestId><HostId>d1d50041-hn-1-hn</HostId></Error> I enable the debug_rgw and what I can see is really strange. the domain has been added :8084, so it make "canonical request hash" and "signature" between client and server unmatched. I can't explain why does this happens 2023-06-29T17:10:46.880+0700 7f26014b0700 10 v4 credential format = DNMZAFE6G2PP8H9P05UU/20230629/us-east-1/s3/aws4_request 2023-06-29T17:10:46.880+0700 7f26014b0700 10 access key id = DNMZAFE6G2PP8H9P05UU 2023-06-29T17:10:46.880+0700 7f26014b0700 10 credential scope = 20230629/us-east-1/s3/aws4_request 2023-06-29T17:10:46.880+0700 7f26014b0700 10 req 15647562574720867919 1000005ns canonical headers format = host:hn.ss.bfcplatform.vn:8084 2023-06-29T17:10:46.880+0700 7f26014b0700 10 payload request hash = UNSIGNED-PAYLOAD 2023-06-29T17:10:46.880+0700 7f26014b0700 10 canonical request = PUT /huynnp-testbucket1/testfile.txt X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=DNMZAFE6G2PP8H9P05UU%2F20230629%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230629T101046Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host host:hn.ss.bfcplatform.vn:8084 host UNSIGNED-PAYLOAD 2023-06-29T17:10:46.880+0700 7f26014b0700 10 canonical request hash = d28e6c3104aff99e9928f902892627d2b284a29d489fbb034ed5c90aa21c566a 2023-06-29T17:10:46.880+0700 7f26014b0700 10 string to sign = AWS4-HMAC-SHA256 20230629T101046Z 20230629/us-east-1/s3/aws4_request d28e6c3104aff99e9928f902892627d2b284a29d489fbb034ed5c90aa21c566a

10 months, 1 week

2
4
0 0

Quincy osd bench in order to define osd_mclock_max_capacity_iops_[hdd|ssd]

by Rafael Diaz Maurin

Hello, I've just upgraded a Pacific cluster into Quincy, and all my osd have the low value osd_mclock_max_capacity_iops_hdd : 315.000000. The manuel does not explain how to benchmark the OSD with fio or ceph bench with good options. Can someone have the good ceph bench options or fio options in order to configure osd_mclock_max_capacity_iops_hdd for each osd ? I ran this bench various times on the same OSD (class hdd) and I obtain different results. ceph tell ${osd} cache drop ceph tell ${osd} bench 12288000 4096 4194304 100 example : osd.21 (hdd): osd_mclock_max_capacity_iops_hdd = 315.000000 bench 1 : 3006.2271379745534 bench 2 : 819.503206458996 bench 3 : 946.5406320134085 How can I succeed in getting the good values for the osd_mclock_max_capacity_iops_[hdd|ssd] options ? Thank you for your help, Rafael

10 months, 1 week

2
2
0 0

Help needed to configure erasure coding LRC plugin

by Michel Jouvin

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

10 months, 1 week

4
20
0 0

ceph-fuse crash

by hakesudu＠gmail.com

Hi, I've deployed a ceph-quincy for HPC. Recently, I always encounter the problem of ceph-fuse crash kernel version is 4.18.0-348.el8.0.2.x86_64 here is part of ceph-fuse log: -59> 2023-06-28T09:51:00.452+0800 155546ff7700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -58> 2023-06-28T09:51:00.452+0800 15554cc49700 3 client.159239 ll_opendir 0x10003e1408d.head -57> 2023-06-28T09:51:00.452+0800 15554cc49700 3 client.159239 may_open 0x1554e79123d0 = 0 -56> 2023-06-28T09:51:00.452+0800 15554cc49700 3 client.159239 ll_opendir 0x10003e1408d.head = 0 (0x155328079380) -55> 2023-06-28T09:51:00.453+0800 1555473f9700 3 client.159239 seekdir(0x155328079380, 0) -54> 2023-06-28T09:51:00.452+0800 155546bf5700 5 client.159239 put_cap_ref dropped last FILE_CACHE ref on 0x20004a6e548.head(faked_ino=0 nref=14 ll_ref=2 cap_refs={1024=0,2048=1} open={1=1} mode=100644 size=5626/0 nlink=1 btime=2023-06-05T14:38:36.471178+0800 mtime=2023-06-05T14:38:36.471178+0800 ctime=2023-06-05T14:38:36.471178+0800 change_attr=1 caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[0x20004a6e548 ts 0/0 objects 1 dirty_or_tx 0] 0x1554e7902300) -53> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 ll_read 0x15531806b970 0~8192 = 5626 -52> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 may_lookup 0x155420004840 = 0 -51> 2023-06-28T09:51:00.453+0800 1554a5dee700 3 client.159239 ll_lookup 0x10003e1406f.head html -50> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -> 0 (100019b7a43) -49> 2023-06-28T09:51:00.453+0800 1554a5dee700 3 client.159239 may_lookup 0x1554c1a89e40 = 0 -48> 2023-06-28T09:51:00.453+0800 1554a7dfe700 3 client.159239 ll_flush 0x15531806b970 0x20004a6e548 -47> 2023-06-28T09:51:00.453+0800 1555469f4700 3 client.159239 ll_lookup 0x100019b7a43.head envs -46> 2023-06-28T09:51:00.453+0800 1555469f4700 3 client.159239 may_lookup 0x155420343eb0 = 0 -45> 2023-06-28T09:51:00.453+0800 1555469f4700 3 client.159239 ll_lookup 0x100019b7a43.head envs -> 0 (200035de5d8) -44> 2023-06-28T09:51:00.453+0800 155545fef700 3 client.159239 ll_release (fh)0x15531806b970 0x20004a6e548 -43> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 seekdir(0x15544c054010, 1152360438801891331) -42> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 seekdir(0x15544c029ee0, 1152690945930559491) -41> 2023-06-28T09:51:00.453+0800 15554cc49700 3 client.159239 ll_releasedir 0x15544c029ee0 -40> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x20004a66459.head tests -39> 2023-06-28T09:51:00.453+0800 15554c244700 3 client.159239 ll_lookup 0x100040bd04e.head att5410-w -> 0 (200047cbc02) -38> 2023-06-28T09:51:00.453+0800 1554a7dfe700 3 client.159239 ll_lookup 0x200035de5d8.head steven-colossal -37> 2023-06-28T09:51:00.453+0800 1554a7dfe700 3 client.159239 may_lookup 0x1554c20229c0 = 0 -36> 2023-06-28T09:51:00.453+0800 1554a7dfe700 3 client.159239 ll_lookup 0x200035de5d8.head steven-colossal -> 0 (100040bcf3a) -35> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 may_lookup 0x155533c970f0 = 0 -34> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x20004a66459.head tests -> 0 (20004a6e3e0) -33> 2023-06-28T09:51:00.453+0800 155545bed700 3 client.159239 ll_releasedir 0x15544c054010 -32> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 ll_getattr 0x200047cbc02.head = 0 -31> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -30> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 may_lookup 0x155420004840 = 0 -29> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -> 0 (100019b7a43) -28> 2023-06-28T09:51:00.453+0800 155545fef700 3 client.159239 ll_lookup 0x100040bcf3a.head lib -27> 2023-06-28T09:51:00.453+0800 155545fef700 3 client.159239 may_lookup 0x1554d7d1d240 = 0 -26> 2023-06-28T09:51:00.453+0800 155545fef700 3 client.159239 ll_lookup 0x100040bcf3a.head lib -> 0 (100040bd018) -25> 2023-06-28T09:51:00.453+0800 1555469f4700 3 client.159239 ll_lookup 0x20004a6e3e0.head test_trainer -24> 2023-06-28T09:51:00.453+0800 1555469f4700 3 client.159239 may_lookup 0x155531e5a6c0 = 0 -23> 2023-06-28T09:51:00.453+0800 15554cc49700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -22> 2023-06-28T09:51:00.453+0800 15554cc49700 3 client.159239 may_lookup 0x155420004840 = 0 -21> 2023-06-28T09:51:00.453+0800 15554cc49700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -> 0 (100019b7a43) -20> 2023-06-28T09:51:00.453+0800 15554c244700 3 client.159239 ll_lookup 0x100019b7a43.head envs -19> 2023-06-28T09:51:00.453+0800 15554c244700 3 client.159239 may_lookup 0x155420343eb0 = 0 -18> 2023-06-28T09:51:00.453+0800 15554c244700 3 client.159239 ll_lookup 0x100019b7a43.head envs -> 0 (200035de5d8) -17> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 ll_lookup 0x100019b7a43.head envs -16> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 may_lookup 0x155420343eb0 = 0 -15> 2023-06-28T09:51:00.453+0800 155546ff7700 3 client.159239 ll_lookup 0x100019b7a43.head envs -> 0 (200035de5d8) -14> 2023-06-28T09:51:00.453+0800 155545bed700 3 client.159239 ll_lookup 0x100040bd018.head terminfo -13> 2023-06-28T09:51:00.453+0800 1555475fa700 3 client.159239 ll_lookup 0x1000383ca0a.head train.py -> 0 (1000383caf7) -12> 2023-06-28T09:51:00.453+0800 1555465f2700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -11> 2023-06-28T09:51:00.453+0800 1555465f2700 3 client.159239 may_lookup 0x155420004840 = 0 -10> 2023-06-28T09:51:00.453+0800 1555465f2700 3 client.159239 ll_lookup 0x200017f674a.head anaconda3 -> 0 (100019b7a43) -9> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x200035de5d8.head steven-colossal -8> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 may_lookup 0x1554c20229c0 = 0 -7> 2023-06-28T09:51:00.453+0800 1555452da700 3 client.159239 ll_lookup 0x200035de5d8.head steven-colossal -> 0 (100040bcf3a) -6> 2023-06-28T09:51:00.453+0800 15554cc49700 3 client.159239 ll_lookup 0x100019b7a43.head envs -5> 2023-06-28T09:51:00.453+0800 1555465f2700 3 client.159239 ll_getattr 0x1000383caf7.head = 0 -4> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 ll_lookup 0x200035de5d8.head llmzoo -3> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 may_lookup 0x1554c20229c0 = 0 -2> 2023-06-28T09:51:00.453+0800 155546bf5700 3 client.159239 ll_lookup 0x200035de5d8.head llmzoo -> 0 (10003e11a9e) -1> 2023-06-28T09:51:00.453+0800 15554c646700 3 client.159239 ll_lookup 0x10003e11a9e.head lib 0> 2023-06-28T09:51:00.458+0800 1554a77fb700 -1 *** Caught signal (Segmentation fault) ** in thread 1554a77fb700 thread_name:ceph-fuse ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: /lib64/libpthread.so.0(+0x12ce0) [0x1555535eece0] 2: (Client::_readdir_cache_cb(dir_result_t*, int (*)(void*, dirent*, ceph_statx*, long, Inode*), void*, int, bool)+0x2f4) [0x555555647d64] 3: (Client::readdir_r_cb(dir_result_t*, int (*)(void*, dirent*, ceph_statx*, long, Inode*), void*, unsigned int, unsigned int, bool)+0xae7) [0x55555564cd37] 4: ceph-fuse(+0xadbf8) [0x555555601bf8] 5: /lib64/libfuse.so.2(+0x16706) [0x1555550fd706] 6: /lib64/libfuse.so.2(+0x17868) [0x1555550fe868] 7: /lib64/libfuse.so.2(+0x14440) [0x1555550fb440] 8: /lib64/libpthread.so.0(+0x81cf) [0x1555535e41cf] 9: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 1/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/ 5 rgw_datacache 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite 0/ 5 seastore 0/ 5 seastore_onode 0/ 5 seastore_odata 0/ 5 seastore_omap 0/ 5 seastore_tm 0/ 5 seastore_cleaner 0/ 5 seastore_lba 0/ 5 seastore_cache 0/ 5 seastore_journal 0/ 5 seastore_device 0/ 5 alienstore 1/ 5 mclock 1/ 5 ceph_exporter -2/-2 (syslog threshold) -1/-1 (stderr threshold) --- pthread ID / name mapping for recent threads --- 1554947e3700 / ceph-fuse 155494fe7700 / ceph-fuse 1554975fa700 / ceph-fuse 1554a57eb700 / ceph-fuse 1554a59ec700 / ceph-fuse 1554a5dee700 / ceph-fuse 1554a5fef700 / ceph-fuse 1554a61f0700 / 1554a63f1700 / ceph-fuse 1554a65f2700 / ceph-fuse 1554a67f3700 / ceph-fuse 1554a6bf5700 / ceph-fuse 1554a6df6700 / ceph-fuse 1554a71f8700 / ceph-fuse 1554a73f9700 / ceph-fuse 1554a75fa700 / 1554a77fb700 / ceph-fuse 1554a7dfe700 / ceph-fuse 1554a7fff700 / ceph-fuse 1555452da700 / ceph-fuse 1555455ea700 / ceph-fuse 1555459ec700 / ceph-fuse 155545bed700 / ceph-fuse 155545fef700 / ceph-fuse 1555465f2700 / ceph-fuse 1555469f4700 / ceph-fuse 155546bf5700 / ceph-fuse 155546df6700 / ceph-fuse 155546ff7700 / ceph-fuse 1555471f8700 / ceph-fuse 1555473f9700 / ceph-fuse 1555475fa700 / ceph-fuse 1555477fb700 / 155547dfe700 / ceph-fuse 155547fff700 / ceph-fuse 15554c244700 / ceph-fuse 15554c646700 / ceph-fuse 15554c847700 / ceph-fuse 15554cc49700 / ceph-fuse 15554ce4a700 / ceph-fuse 15554da50700 / ms_dispatch max_recent 10000 max_new 10000 log_file /var/lib/ceph/crash/2023-06-28T01:51:00.459615Z_3ddbaa44-d8cd-437b-908b-c3772520c7a6/log --- end dump of recent events --- Has anyone encountered this kind of problem?

10 months, 1 week

2
1
0 0

Quincy osd bench in order to define osd_mclock_max_capacity_iops_[hdd|ssd]

by Rafael Diaz Maurin

Hello, I've just upgraded a Pacific cluster into Quincy, and all my osd have the low value osd_mclock_max_capacity_iops_hdd : 315.000000. The manuel does not explain how to benchmark the OSD with fio or ceph bench with good options. Can someone have the good ceph bench options or fio options in order to configure osd_mclock_max_capacity_iops_hdd for each osd ? I ran this bench various times on the same OSD (class hdd) and I obtain different results. ceph tell ${osd} cache drop ceph tell ${osd} bench 12288000 4096 4194304 100 example : osd.21 (hdd): osd_mclock_max_capacity_iops_hdd = 315.000000 bench 1 : 3006.2271379745534 bench 2 : 819.503206458996 bench 3 : 946.5406320134085 How can I succeed in getting the good values for the osd_mclock_max_capacity_iops_[hdd|ssd] options ? Thank you for your help, Rafael

10 months, 1 week

1
0
0 0

warning: CEPHADM_APPLY_SPEC_FAIL

by Adiga, Anantha

Hi, I am not finding any reference to clear this warning AND stop the service. See below After creating OSD with iops_optimized option, this WARN mesg appear. Ceph 17.2.6 [cid:image001.png@01D9AAA5.8639A1F0] 6/29/23 4:10:45 PM [WRN] Health check failed: Failed to apply 1 service(s): osd.iops_optimized (CEPHADM_APPLY_SPEC_FAIL) 6/29/23 4:10:45 PM [ERR] Failed to apply osd.iops_optimized spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: iops_optimized service_name: osd.iops_optimized placement: host_pattern: '*' spec: data_devices: rotational: 0 filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/mon.fl31ca104ja0203/config Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e -e NODE_NAME=fl31ca104ja0203 -e #:/var/log/ceph# grep optimi cephadm.log cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=iops_optimized', '--image', 'quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e', 'ceph-volume', '--fsid', 'd0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/nvme10n1', '/dev/nvme11n1', '/dev/nvme12n1', '/dev/nvme13n1', '/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1', '/dev/nvme5n1', '/dev/nvme6n1', '/dev/nvme7n1', '/dev/nvme8n1', '/dev/nvme9n1', '--yes', '--no-systemd'] 2023-06-29 23:06:28,340 7fc2668a7740 INFO Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e -e NODE_NAME=fl31ca104ja0202 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=iops_optimized -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e:/var/run/ceph:z -v /var/log/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e:/var/log/ceph:z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp1v09i0jx:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphy3pnh46:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e lvm batch --no-auto /dev/nvme10n1 /dev/nvme11n1 /dev/nvme12n1 /dev/nvme13n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1 /dev/nvme6n1 /dev/nvme7n1 /dev/nvme8n1 /dev/nvme9n1 --yes --no-systemd #:/var/log/ceph# grep optimi cephadm.log cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=iops_optimized', '--image', 'quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e', 'ceph-volume', '--fsid', 'd0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', '/dev/nvme10n1', '/dev/nvme11n1', '/dev/nvme12n1', '/dev/nvme13n1', '/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1', '/dev/nvme5n1', '/dev/nvme6n1', '/dev/nvme7n1', '/dev/nvme8n1', '/dev/nvme9n1', '--yes', '--no-systemd'] 2023-06-29 23:06:28,340 7fc2668a7740 INFO Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e<mailto:CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e> -e NODE_NAME=fl31ca104ja0202 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=iops_optimized -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e:/var/run/ceph:z -v /var/log/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e:/var/log/ceph:z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp1v09i0jx:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphy3pnh46:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e<mailto:quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e> lvm batch --no-auto /dev/nvme10n1 /dev/nvme11n1 /dev/nvme12n1 /dev/nvme13n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1 /dev/nvme6n1 /dev/nvme7n1 /dev/nvme8n1 /dev/nvme9n1 --yes --no-systemd The warning message clears on its on for a few seconds..ceph health status foes green, but comes back. I am not finding any reference to clear this msg AND not able to stop the service. /#ceph orch ls node-exporter ?:9100 4/4 8m ago 10w * osd 33 8m ago - <unmanaged> osd.iops_optimized 0 - 2d * /# ceph orch stop --service_name osd.iops_optimized Error EINVAL: No daemons exist under service name "osd.iops_optimized". View currently running services using "ceph orch ls" /# Thank you, Anantha

10 months, 2 weeks

1
1
0 0

1 pg inconsistent and does not recover

by Niklas Hambüchen

Hi, I have a 3x-replicated pool with Ceph 12.2.7. One HDD broke, its OSD "2" was automatically marked as "out", the disk was physically replaced by a new one, and that added back in. Now `ceph health detail` continues to permanently show: [ERR] OSD_SCRUB_ERRORS: 1 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 2.87 is active+clean+inconsistent, acting [33,2,20] What exactly is wrong here? Why can Ceph not fix the issue? With BlueStore I have checksums, on two unbroken disks, so what remaining inconsistency can there be? The suggested command in https://docs.ceph.com/en/pacific/rados/operations/pg-repair/#commands-for-d… does not work: # rados list-inconsistent-obj 2.87 No scrub information available for pg 2.87 error 2: (2) No such file or directory Further, I find the documentation in https://docs.ceph.com/en/pacific/rados/operations/pg-repair/#more-informati… extremely unclear. It says > In the case of replicated pools, recovery is beyond the scope of pg repair. while many people on the Internet suggest that `ceph pg repair` might fix the issue. Yet again others claim that Ceph will fix the issue itself. I am hesitant to run "ceph pg repair" without understanding what the problem is and what exactly this will do. I have already reported the "error 2" and the documentation in issue https://tracker.ceph.com/issues/61739 but not received a reply yet, and my cluster stays "inconsistent". How can this be fixed? I would appreciate any help!

10 months, 2 weeks

5
15
0 0

device class for nvme disk is ssd

by Boris Behrens

Hi, is it a problem that the device class for all my disks is SSD even all of these disks are NVME disks? If it is just a classification for ceph, so I can have pools on SSDs and NVMEs separated I don't care. But maybe ceph handles NVME disks differently internally? I've added them via ceph-volume lvm create --bluestore --data /dev/nvme2n1 and they only show up as ssd root@a0423f621aaa:~# ceph osd metadata osd.0 { "id": 0, "arch": "x86_64", ... "bluefs": "1", "bluefs_dedicated_db": "0", "bluefs_dedicated_wal": "0", "bluefs_single_shared_device": "1", "bluestore_bdev_access_mode": "blk", "bluestore_bdev_block_size": "4096", "bluestore_bdev_dev_node": "/dev/dm-2", "bluestore_bdev_devices": "nvme0n1", "bluestore_bdev_driver": "KernelDevice", "bluestore_bdev_partition_path": "/dev/dm-2", "bluestore_bdev_rotational": "0", "bluestore_bdev_size": "1920378863616", "bluestore_bdev_support_discard": "1", "bluestore_bdev_type": "ssd", "ceph_release": "pacific", "ceph_version": "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)", "ceph_version_short": "16.2.13", "ceph_version_when_created": "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)", "cpu": "Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz", "created_at": "2023-06-20T14:03:35.167741Z", "default_device_class": "ssd", "device_ids": "nvme0n1=SAMSUNG_MZQLB1T9HAJR-00007_S439NF0M506164", "device_paths": "nvme0n1=/dev/disk/by-path/pci-0000:5e:00.0-nvme-1", "devices": "nvme0n1", "distro": "ubuntu", "distro_description": "Ubuntu 20.04.6 LTS", "distro_version": "20.04", ... "journal_rotational": "0", "kernel_description": "#169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023", "kernel_version": "5.4.0-152-generic", "mem_swap_kb": "0", "mem_total_kb": "196668116", "network_numa_unknown_ifaces": "back_iface,front_iface", "objectstore_numa_node": "0", "objectstore_numa_nodes": "0", "os": "Linux", "osd_data": "/var/lib/ceph/osd/ceph-0", "osd_objectstore": "bluestore", "osdspec_affinity": "", "rotational": "0" } Cheers Boris

10 months, 2 weeks

5
6
0 0

CLT Meeting Notes June 28th, 2023

by Adam King

Reef RC linking failure on Alpine Linux. Do we worry about that? 1. https://tracker.ceph.com/issues/61718 2. Nice to fix, but not a requirement 3. If there are patches available, we should accept them, but probably don't put too much work into it currently debian bullseye build failure on reef rc: 1. https://tracker.ceph.com/issues/61845 2. want to fix before final release clean-up in AuthMonitor – CephFS and core are fine. Any other component interested in? 1. https://github.com/ceph/ceph/pull/52008#issuecomment-1606581139 2. Already tested for rados and cephfs 3. No other components requesting testing Reef rc v18.1.2 in progress 1. Next rc build in progress 2. build issue to be looked at 1. Failure on a jammy arm build, not a platform we test on meaningfully 2. Think this is an infrastructure issue 3. Generally, this shouldn't be a release blocker 4. Priority of arm builds might rise in the future though 5. investigate today, if we can't figure it out quickly, publish rc with a known issue 6. NOTE: Rook expects arm builds to be present 3. Would like to release next rc later this week if things work out 4. Would also like to upgrade lrc CDS agenda https://pad.ceph.com/p/cds-squid 1. leads should add topics 2. plan is for this to happen week of July 17th mempool monitoring in teuthology tests https://github.com/ceph/ceph/pull/51853 1. Just an FYI 2. ceph task will now have ability to dump memtools 3. might add a bit of delay to how long tests take 4. expected to be merged soon 5. will follow up in performance meeting iSCSI packages old/not signed -- want to fix before final release 1. https://tracker.ceph.com/issues/57388 2. tcmu-runner, since containerization of ceph, is being pulled from our build system 3. This tcmu-runner package is not signed 4. ceph-iscsi package is signed, but outdated (seems to be because this is the newest one that is signed and pushed to download.ceph.com) 5. someone with access to tools to sign the packages would have to help fix this 6. been like this for a long time and nobody noticed 7. only ceph-iscsi package, not tcmu-runner, is distributed through download.ceph.com 8. getting updated ceph-iscsi package on download.ceph.com should be done before reef release 9. tcmu-runner inside the container being unsigned is not as big of a deal (was this way in quincy/pacific as well)

10 months, 2 weeks

1
0
0 0

Re: [multisite] period update and zonegroup

by Yixin Jin

Mystery solved. I loaded the zonegroup up with 100K hostnanes. That caused the period to fail to decode when other zones try to pick up the new period. Apparently, there is a size limit on how big zonegroup info can be but implicitly. Regards, Yixin Sent from Yahoo Mail on Android

10 months, 2 weeks

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users June 2023