- ceph-users - lists.ceph.io

Does ceph support CentOS8/RHEL8 now?
by FuLong Wang 18 Apr '20

18 Apr '20

Hi Everyone, We want to deploy ceph on CentOS8/RHEL8 environment, is it supported now? From below link, i see only CentOS7/RHEL7 supported, right? https://docs.ceph.com/docs/master/start/os-recommendations/ -- FuLong Wang _______________________________________________

1 0

Understanding PG peering count increase without osdmap changes
by Prasad Krishnan 18 Apr '20

18 Apr '20

Hi, I am trying to understand how the PG peering count could increase between two OSDMAP epochs when none of the OSDs went down or up, nor were there changes in their weights. This behaviour was seen on a Hammer cluster, but I guess it should be no different from the recent releases. I'm pasting the relevant piece of monitor logs below and attaching the relevant odmap dumps. As it can be seen, the changes to the osdmap between dump_e1053237 and dump_e10532398 seem to be restricted to a few pg_temp additions and changes to up_thru, but the OSD weights and OSD up/down state have remained the same. Kindly let me know what I'm missing here in my understanding. Thanks, K.Prasad Monitor logs -------------------- 2020-04-15 08:06:10.956681 mon.1 10.33.89.159:6789/0 52232955 : cluster [INF] pgmap v107549051: 18432 pgs: 18332 active+clean, 1 peering, 99 active+remapped+backfill_toofull; 330 TB data, 518 TB used, 596 TB / 1114 TB avail; 34776/1684702815 objects degraded (0.002%); 1766213/1684702815 objects misplaced (0.105%) 2020-04-15 08:06:11.862613 mon.1 10.33.89.159:6789/0 52232956 : cluster [INF] osdmap e1053236: 388 osds: 315 up, 308 in 2020-04-15 08:06:11.868380 mon.1 10.33.89.159:6789/0 52232957 : cluster [INF] pgmap v107549052: 18432 pgs: 18332 active+clean, 1 peering, 99 active+remapped+backfill_toofull; 330 TB data, 518 TB used, 596 TB / 1114 TB avail; 34776/1684702815 objects degraded (0.002%); 1766213/1684702815 objects misplaced (0.105%) 2020-04-15 08:06:10.460979 osd.57 10.33.249.161:6811/4066471 200448 : cluster [INF] 5.356s0 restarting backfill on osd.382(0) from (936954'62108,945456'62308] 0//0//-1 to 1053229'64769 2020-04-15 08:06:12.696839 mon.1 10.33.89.159:6789/0 52232959 : cluster [INF] osdmap e1053237: 388 osds: 315 up, 308 in 2020-04-15 08:06:12.703424 mon.1 10.33.89.159:6789/0 52232960 : cluster [INF] pgmap v107549053: 18432 pgs: 18332 active+clean, 1 peering, 99 active+remapped+backfill_toofull; 330 TB data, 518 TB used, 596 TB / 1114 TB avail; 34776/1684702815 objects degraded (0.002%); 1766213/1684702815 objects misplaced (0.105%) 2020-04-15 08:06:13.745554 mon.1 10.33.89.159:6789/0 52232961 : cluster [INF] osdmap e1053238: 388 osds: 315 up, 308 in 2020-04-15 08:06:13.899273 mon.1 10.33.89.159:6789/0 52232962 : cluster [INF] pgmap v107549054: 18432 pgs: 1 inactive, 1 active+remapped+wait_backfill+backfill_toofull, 2 activating, 17078 active+clean, 1251 peering, 2 activating+remapped, 5 remapped+peering, 1 activating+degraded, 91 active+remapped+backfill_toofull; 329 TB data, 518 TB used, 600 TB / 1118 TB avail; 32298/1678972195 objects degraded (0.002%); 1666716/1678972195 objects misplaced (0.099%) 2020-04-15 08:06:14.766039 mon.1 10.33.89.159:6789/0 52232963 : cluster [INF] osdmap e1053239: 388 osds: 315 up, 308 in 2020-04-15 08:06:14.833707 mon.1 10.33.89.159:6789/0 52232964 : cluster [INF] pgmap v107549055: 18432 pgs: 2 inactive, 1 active+remapped+wait_backfill+backfill_toofull, 3 activating, 16496 active+clean, 1794 peering, 4 activating+remapped, 42 remapped+peering, 3 activating+degraded, 87 active+remapped+backfill_toofull; 329 TB data, 518 TB used, 600 TB / 1118 TB avail; 30842/1678959760 objects degraded (0.002%); 1576530/1678959760 objects misplaced (0.094%) 2020-04-15 08:06:15.730301 mon.1 10.33.89.159:6789/0 52232965 : cluster [INF] osdmap e1053240: 388 osds: 315 up, 308 in 2020-04-15 08:06:15.774021 mon.1 10.33.89.159:6789/0 52232966 : cluster [INF] pgmap v107549056: 18432 pgs: 2 inactive, 1 active+remapped+wait_backfill+backfill_toofull, 3 activating, 16139 active+clean, 2103 peering, 6 activating+remapped, 92 remapped+peering, 3 activating+degraded, 83 active+remapped+backfill_toofull; 329 TB data, 518 TB used, 600 TB / 1118 TB avail; 29414/1678947337 objects degraded (0.002%); 1490753/1678947337 objects misplaced (0.089%) 2020-04-15 08:06:09.424940 osd.146 10.33.129.28:6800/2689 125912 : cluster [INF] 5.252fs0 restarting backfill on osd.382(3) from (935430'62507,945456'62707] 0//0//-1 to 1053229'65086 2020-04-15 08:06:16.713440 mon.1 10.33.89.159:6789/0 52232967 : cluster [INF] osdmap e1053241: 388 osds: 315 up, 308 in -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

1 0

HELP! Ceph( v 14.2.8) bucket notification dose not work!
by 曹海旺 17 Apr '20

17 Apr '20

Hi, I upgrade the ceph from 14.2.7 to the new version 14.2.8 . The bucket notification dose not work. I can’t create a TOPIC : I use post man to send a post flow by https://docs.ceph.com/docs/master/radosgw/notifications/#create-a-topic REQUEST: POST http://rgw1:7480/?Action=CreateTopic&Name=webno&push-endpoint=https://192.1… RESPONSE: <?xml version="1.0" encoding="UTF-8"?> <Error> <Code>MethodNotAllowed</Code> <RequestId>tx000000000000000000008-005e6a0eab-cbcad-bj</RequestId> <HostId>cbcad-bj-bjz</HostId> </Error> The debug info on the node below: 2020-03-12 18:49:24.684 7fdde1e1d700 1 ====== starting new request req=0x55c91a51e8f0 ===== 2020-03-12 18:49:24.684 7fdde1e1d700 2 req 14 0.000s initializing for trans_id = tx00000000000000000000e-005e6a13b4-cbc6a-bj 2020-03-12 18:49:24.684 7fdde1e1d700 10 rgw api priority: s3=8 s3website=7 2020-03-12 18:49:24.684 7fdde1e1d700 10 host=192.168.3.250 2020-03-12 18:49:24.684 7fdde1e1d700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2020-03-12 18:49:24.684 7fdde1e1d700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/ 2020-03-12 18:49:24.684 7fdde1e1d700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256 2020-03-12 18:49:24.684 7fdde1e1d700 10 meta>> HTTP_X_AMZ_DATE 2020-03-12 18:49:24.684 7fdde1e1d700 10 x>> x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2020-03-12 18:49:24.684 7fdde1e1d700 10 x>> x-amz-date:20200312T104924Z 2020-03-12 18:49:24.684 7fdde1e1d700 20 get_handler handler=26RGWHandler_REST_Service_S3 2020-03-12 18:49:24.684 7fdde1e1d700 10 handler=26RGWHandler_REST_Service_S3 2020-03-12 18:49:24.684 7fdde1e1d700 2 req 14 0.000s getting op 4 2020-03-12 18:49:24.684 7fdde1e1d700 20 handler->ERRORHANDLER: err_no=-2003 new_err_no=-2003 2020-03-12 18:49:24.684 7fdde1e1d700 2 req 14 0.000s http status=405 2020-03-12 18:49:24.684 7fdde1e1d700 1 ====== req done req=0x55c91a51e8f0 op status=0 http_status=405 latency=0s ====== 2020-03-12 18:49:25.502 7fde0fe79700 2 RGWDataChangesLog::ChangesRenewThread: start The same post works in the version 14.2.7 What is the correct way to create a Topic in version 14.2.8 ？

3 3

Re: RGW and the orphans
by Eric Ivancich 17 Apr '20

17 Apr '20

> On Apr 16, 2020, at 1:58 PM, EDH - Manuel Rios <mriosfer(a)easydatahost.com> wrote: > > Hi Eric, > > Are there any ETA for get those script backported maybe in 14.2.10? > > Regards > Manuel There is a nautilus backport PR where the code works. It’s waiting on the added testing to be complete on master, so that can be backported as well. See: https://github.com/ceph/ceph/pull/34127 <https://github.com/ceph/ceph/pull/34127> -- J. Eric Ivancich he / him / his Red Hat Storage Ann Arbor, Michigan, US

1 0

radosgw-admin error: "could not fetch user info: no user info saved"
by Mathew Snyder 17 Apr '20

17 Apr '20

I'm running into a problem that I've found around the Internet, but for which I'm unable to find a solution: $ sudo radosgw-admin user info could not fetch user info: no user info saved I've seen others report this issue, but the troubleshooting has either failed to pinpoint the issue, or has gone from "I have a problem" to "I solved it" with no information as to how it was solved. Unfortunately, due to the nature of our network, I'm unable to provide debug information en masse. I can try to provide individual lines of output if someone knows what I should look for. I am able to run the user creation command: $ sudo radosgw-admin user create --uid="testuser2" --display-name="testuser2" { "user_id": "testuser2", "display_name": "testuser2", "email": "", "suspended": 0, "max_buckets": 1000, ...<snip>.... } After doing this, I'm still unable to see the user information instead getting the same "no user info saved" output. That said, I know the user is added: $ sudo radosgw-admin metadata list user [ "devtest1", "testuser", "testuser2", ...<snip>... ] I'm pretty new to this system and Ceph in general. I looked at /etc/ceph/ceph.conf but I don't see anything that would have anything to do with users. I also don't have an account on the Ansible server that controls the configurations so I can't see if anything is defined there. I know this isn't particularly detailed in its description of the issue, but I can't find any other useful help via search. Thank you, Mathew Sent with [ProtonMail](https://protonmail.com) Secure Email.

2 3

Re: PGs unknown (osd down) after conversion to cephadm
by Dr. Marco Savoca 17 Apr '20

17 Apr '20

1 0

MDS: what's the purpose of using LogEvent with empty metablob?
by Xinying Song 17 Apr '20

17 Apr '20

Hi, cephers: What's the purpose of using LogEvent with empty metablob? For example in link/unlink operation cross two active mds, when slave receives OP_FINISH it will write an ESlaveUpdate::OP_COMMIT to the journal, then send OP_COMMITTED to master. When master receives OP_COMMITTED it will write an ECommitted to the journal then allow previously logged journal to be trimmed. Why are these two logevents necessary? I guess they are originally used for a scene that crashes happen, but in my opinion it seems not necessary. For example, if cash happens, after failed mds are brought up again, in resolve stage, master will resend OP_FINISH to slave, then things will continue as expected. Could anyone give some tips on this doubt? Sincerely thanks!

3 6

how to fix num_strays?
by Dan van der Ster 17 Apr '20

17 Apr '20

Hi all, Following some cephfs issues today we have a stable cluster but the num_strays is incorrect. After starting the mds, the values are reasonable, but they very soon underflow and start showing 18E (2^64 - a few) ---------------mds---------------- --mds_cache--- ------mds_log------ -mds_mem- ----mds_server----- mds_ ---objecter--- req rlat fwd inos caps exi imi |stry recy recd|subm evts segs repl|ino dn |hcr hcs hsr cre |sess|actv rd wr | 129 0 0 253k 1.9k 0 0 |246 0 0 | 5 4.0k 5 0 |253k 254k|129 0 0 0 |119 | 3 1 2 8.2k 0 0 253k 1.9k 0 0 |129 0 0 |395 4.4k 7 0 |253k 254k|8.2k 11 0 0 |119 | 0 33 517 9.7k 0 0 253k 1.8k 0 0 |181 0 0 |302 4.7k 7 0 |253k 254k|9.7k 5 0 0 |119 | 1 44 297 10k 0 0 253k 1.8k 0 0 |217 0 0 |382 5.1k 7 0 |253k 254k| 10k 11 0 0 |119 | 0 54 405 9.0k 0 0 253k 1.7k 0 0 |205 0 0 |386 5.5k 8 0 |253k 254k|9.0k 4 0 0 |119 | 1 46 431 8.2k 0 0 253k 1.7k 0 0 |161 0 0 |326 5.8k 8 0 |253k 254k|8.2k 6 0 0 |119 | 1 37 397 8.0k 0 0 253k 1.6k 0 0 |135 0 0 |279 6.1k 8 0 |253k 254k|8.0k 4 0 0 |119 | 1 31 317 9.2k 0 0 253k 1.6k 0 0 | 18E 0 0 |153 6.2k 8 0 |253k 254k|9.2k 6 0 0 |119 | 1 2 265 8.2k 0 0 253k 1.7k 0 0 | 18E 0 0 | 40 6.3k 8 0 |253k 254k|8.2k 5 0 0 |119 | 3 3 17 Is there a way to reset the num_strays to the correct number of strays ? # for i in `seq 600 609`; do rados --cluster=dwight listomapvals -p cephfs_metadata ${i}.00000000 2>&1 | wc -l; done 95 220 91 173 567 1578 1042 754 445 261 Cheers, Dan

2 5

cephadm adopting osd failed
by bbk＠nocloud.ch 17 Apr '20

17 Apr '20

Hi, From a healty nautilus cluster version 14.2.9 on a CentOS7 i try to follow the upgrade procedure to the containerized octopus setup with cephadm. * https://docs.ceph.com/docs/octopus/cephadm/adoption/ Everything step went fine until i wanted to adopt the osds, then i get a error. Does anybody have a idea, what my problem could be? ```````````````````` ~# cephadm adopt --style legacy --name osd.0 INFO:cephadm:Pulling latest docker.io/ceph/ceph:v15 container... INFO:cephadm:objectstore_type is bluestore INFO:cephadm:Stopping old systemd unit ceph-osd@0... INFO:cephadm:Disabling old systemd unit ceph-osd@0... INFO:cephadm:Moving data... Traceback (most recent call last): File "/usr/sbin/cephadm", line 4282, in <module> r = args.func() File "/usr/sbin/cephadm", line 972, in _default_image return func() File "/usr/sbin/cephadm", line 2916, in command_adopt command_adopt_ceph(daemon_type, daemon_id, fsid); File "/usr/sbin/cephadm", line 2979, in command_adopt_ceph os.rmdir(data_dir_src) OSError: [Errno 39] Directory not empty: '//var/lib/ceph/osd/ceph-0' ```````````````````` Yours, bbk

3 5

Re: PGs unknown (osd down) after conversion to cephadm
by Dr. Marco Savoca 16 Apr '20

16 Apr '20

2 1