- ceph-users - lists.ceph.io

Unable to start MDS and access CephFS after upgrade to 17.2.6
by Ben Stöver 15 Jun '23

15 Jun '23

Hi everyone, as discussed on this list before, we had an issue upgrading the metadata servers while performing an upgrade from 17.2.5 to 17.2.6. (See also https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/U3VEPCXYDYO… .) We had to pause the upgrade leaving us in a state where the MDS were running on the old version, while most other cluster components were already running the newer version. In the meantime we were able solve this issue and also reduced the number of active MDS from 2 to 1. Today we resumed the upgrade and all components are on version 17.2.6 now. Right after the upgrade a file system inconsistency (backtrace) was found, which we solved by scrubbing. Shortly after users coming back on the cluster using the CephFS again, all MDS failed and cannot be started again. The first MDS gets stuck in state "replay (laggy)", while both standby MDS crash right away ("Caught signal (Segmentation fault)"). All MDS report multiple of the following problems in the log: - "replayed ESubtreeMap at xxx subtree root 0x1 not in cache" - "replayed ESubtreeMap at xxx subtree root 0x1 is not mine in cache (it's -2,-2)" As a result, we are currently not able to access our CephFS anymore. Does anyone have an idea how to solve this or what could be the cause? (I have found this ticket, which seems to be very similar but there does not seem to be a solution mentioned there. https://bugzilla.redhat.com/show_bug.cgi?id=2056935 ) Below are some parts of the MDS logs that seem relevant to us for this issue. We are thankful for any ideas. :-) Best Ben Log excerpt of Active MDS (replay): -140> 2023-06-14T07:51:59.585+0000 7feb588c0700 5 asok(0x55e56e23a000) register_command objecter_requests hook 0x55e56e1ca380 -139> 2023-06-14T07:51:59.585+0000 7feb588c0700 10 monclient: _renew_subs -138> 2023-06-14T07:51:59.585+0000 7feb588c0700 10 monclient: _send_mon_message to mon.ceph-service-01 at v2:131.220.126.65:3300/0 -137> 2023-06-14T07:51:59.585+0000 7feb588c0700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -136> 2023-06-14T07:51:59.585+0000 7feb588c0700 4 mds.0.purge_queue operator(): data pool 3 not found in OSDMap -135> 2023-06-14T07:51:59.585+0000 7feb588c0700 4 mds.0.0 apply_blocklist: killed 0, blocklisted sessions (0 blocklist entries, 0) -134> 2023-06-14T07:51:59.585+0000 7feb588c0700 1 mds.0.172004 handle_mds_map i am now mds.0.172004 -133> 2023-06-14T07:51:59.585+0000 7feb588c0700 1 mds.0.172004 handle_mds_map state change up:standby --> up:replay -132> 2023-06-14T07:51:59.585+0000 7feb588c0700 5 mds.beacon.fdi-cephfs.ceph-service-01.gfwudy set_want_state: up:standby -> up:replay -131> 2023-06-14T07:51:59.585+0000 7feb588c0700 1 mds.0.172004 replay_start -130> 2023-06-14T07:51:59.585+0000 7feb588c0700 1 mds.0.172004 waiting for osdmap 341143 (which blocklists prior instance) -129> 2023-06-14T07:51:59.585+0000 7feb588c0700 10 monclient: _send_mon_message to mon.ceph-service-01 at v2:131.220.126.65:3300/0 -128> 2023-06-14T07:51:59.585+0000 7feb548b8700 2 mds.0.cache Memory usage: total 300004, rss 34196, heap 182556, baseline 182556, 0 / 0 inodes have caps, 0 caps, 0 caps per inode -127> 2023-06-14T07:51:59.603+0000 7feb588c0700 10 monclient: _renew_subs -126> 2023-06-14T07:51:59.603+0000 7feb588c0700 10 monclient: _send_mon_message to mon.ceph-service-01 at v2:131.220.126.65:3300/0 -125> 2023-06-14T07:51:59.603+0000 7feb588c0700 10 monclient: handle_get_version_reply finishing 1 version 341143 -124> 2023-06-14T07:51:59.603+0000 7feb528b4700 2 mds.0.172004 Booting: 0: opening inotable -123> 2023-06-14T07:51:59.603+0000 7feb528b4700 2 mds.0.172004 Booting: 0: opening sessionmap -122> 2023-06-14T07:51:59.603+0000 7feb528b4700 2 mds.0.172004 Booting: 0: opening mds log -121> 2023-06-14T07:51:59.603+0000 7feb528b4700 5 mds.0.log open discovering log bounds -120> 2023-06-14T07:51:59.603+0000 7feb528b4700 2 mds.0.172004 Booting: 0: opening purge queue (async) -119> 2023-06-14T07:51:59.603+0000 7feb528b4700 4 mds.0.purge_queue open: opening -118> 2023-06-14T07:51:59.603+0000 7feb528b4700 1 mds.0.journaler.pq(ro) recover start -117> 2023-06-14T07:51:59.603+0000 7feb528b4700 1 mds.0.journaler.pq(ro) read_head -116> 2023-06-14T07:51:59.603+0000 7feb520b3700 4 mds.0.journalpointer Reading journal pointer '400.00000000' -115> 2023-06-14T07:51:59.603+0000 7feb528b4700 2 mds.0.172004 Booting: 0: loading open file table (async) -114> 2023-06-14T07:51:59.604+0000 7feb528b4700 2 mds.0.172004 Booting: 0: opening snap table -113> 2023-06-14T07:51:59.604+0000 7feb5c0c7700 10 monclient: get_auth_request con 0x55e56ef7e000 auth_method 0 -112> 2023-06-14T07:51:59.604+0000 7feb5b8c6700 10 monclient: get_auth_request con 0x55e56ef7e800 auth_method 0 -111> 2023-06-14T07:51:59.604+0000 7feb5b0c5700 10 monclient: get_auth_request con 0x55e56ef7f000 auth_method 0 -110> 2023-06-14T07:51:59.604+0000 7feb5c0c7700 10 monclient: get_auth_request con 0x55e56ef92000 auth_method 0 -109> 2023-06-14T07:51:59.605+0000 7feb538b6700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 18366857216, expire 18367220777, write 18367220777, stream_format 1). probing for end of log (from 18367220777)... -108> 2023-06-14T07:51:59.605+0000 7feb538b6700 1 mds.0.journaler.pq(ro) probing for end of the log -107> 2023-06-14T07:51:59.605+0000 7feb520b3700 1 mds.0.journaler.mdlog(ro) recover start -106> 2023-06-14T07:51:59.605+0000 7feb520b3700 1 mds.0.journaler.mdlog(ro) read_head -105> 2023-06-14T07:51:59.605+0000 7feb520b3700 4 mds.0.log Waiting for journal 0x200 to recover... -104> 2023-06-14T07:51:59.605+0000 7feb5b8c6700 10 monclient: get_auth_request con 0x55e56ef7f400 auth_method 0 -103> 2023-06-14T07:51:59.605+0000 7feb5c0c7700 10 monclient: get_auth_request con 0x55e56ef92800 auth_method 0 -102> 2023-06-14T07:51:59.605+0000 7feb5b0c5700 10 monclient: get_auth_request con 0x55e56ef7f800 auth_method 0 -101> 2023-06-14T07:51:59.606+0000 7feb528b4700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 320109193199616, expire 320109196838948, write 320109612836052, stream_format 1). probing for end of log (from 320109612836052)... -100> 2023-06-14T07:51:59.606+0000 7feb528b4700 1 mds.0.journaler.mdlog(ro) probing for end of the log -99> 2023-06-14T07:51:59.606+0000 7feb5b0c5700 10 monclient: get_auth_request con 0x55e56f2ea800 auth_method 0 -98> 2023-06-14T07:51:59.606+0000 7feb5b8c6700 10 monclient: get_auth_request con 0x55e56f2ea000 auth_method 0 -97> 2023-06-14T07:51:59.606+0000 7feb538b6700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 18367220777 (header had 18367220777). recovered. -96> 2023-06-14T07:51:59.606+0000 7feb538b6700 4 mds.0.purge_queue operator(): open complete -95> 2023-06-14T07:51:59.606+0000 7feb538b6700 1 mds.0.journaler.pq(ro) set_writeable -94> 2023-06-14T07:51:59.607+0000 7feb528b4700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 320109613139120 (header had 320109612836052). recovered. -93> 2023-06-14T07:51:59.607+0000 7feb520b3700 4 mds.0.log Journal 0x200 recovered. -92> 2023-06-14T07:51:59.607+0000 7feb520b3700 4 mds.0.log Recovered journal 0x200 in format 1 -91> 2023-06-14T07:51:59.607+0000 7feb520b3700 2 mds.0.172004 Booting: 1: loading/discovering base inodes -90> 2023-06-14T07:51:59.607+0000 7feb520b3700 0 mds.0.cache creating system inode with ino:0x100 -89> 2023-06-14T07:51:59.607+0000 7feb520b3700 0 mds.0.cache creating system inode with ino:0x1 -88> 2023-06-14T07:51:59.608+0000 7feb5c0c7700 10 monclient: get_auth_request con 0x55e56f2eb400 auth_method 0 -87> 2023-06-14T07:51:59.615+0000 7feb528b4700 2 mds.0.172004 Booting: 2: replaying mds log -86> 2023-06-14T07:51:59.615+0000 7feb528b4700 2 mds.0.172004 Booting: 2: waiting for purge queue recovered -85> 2023-06-14T07:51:59.615+0000 7feb5b8c6700 10 monclient: get_auth_request con 0x55e56f1cec00 auth_method 0 -84> 2023-06-14T07:51:59.615+0000 7feb5b0c5700 10 monclient: get_auth_request con 0x55e56f2eb800 auth_method 0 -83> 2023-06-14T07:51:59.615+0000 7feb5c0c7700 10 monclient: get_auth_request con 0x55e56f1ce400 auth_method 0 -82> 2023-06-14T07:51:59.615+0000 7feb5b8c6700 10 monclient: get_auth_request con 0x55e56ef7e400 auth_method 0 -81> 2023-06-14T07:51:59.615+0000 7feb5b0c5700 10 monclient: get_auth_request con 0x55e56f1cf400 auth_method 0 -80> 2023-06-14T07:51:59.634+0000 7feb510b1700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 320109197452549 subtree root 0x1 not in cache -79> 2023-06-14T07:51:59.634+0000 7feb510b1700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -78> 2023-06-14T07:51:59.634+0000 7feb510b1700 0 mds.0.journal journal ambig_subtrees: -77> 2023-06-14T07:51:59.637+0000 7feb510b1700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 320109198067908 subtree root 0x1 not in cache -76> 2023-06-14T07:51:59.637+0000 7feb510b1700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -75> 2023-06-14T07:51:59.637+0000 7feb510b1700 0 mds.0.journal journal ambig_subtrees: -74> 2023-06-14T07:51:59.640+0000 7feb510b1700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 320109198682422 subtree root 0x1 not in cache -73> 2023-06-14T07:51:59.640+0000 7feb510b1700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} Log excerpt of standby MDS: -23> 2023-06-14T07:48:56.590+0000 7f14fcb97700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 320109205645870 subtree root 0x1 is not mine in cache (it's -2,-2) -22> 2023-06-14T07:48:56.590+0000 7f14fcb97700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -21> 2023-06-14T07:48:56.590+0000 7f14fcb97700 0 mds.0.journal journal ambig_subtrees: -20> 2023-06-14T07:48:56.592+0000 7f14fcb97700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 320109206260384 subtree root 0x1 is not mine in cache (it's -2,-2) -19> 2023-06-14T07:48:56.592+0000 7f14fcb97700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -18> 2023-06-14T07:48:56.592+0000 7f14fcb97700 0 mds.0.journal journal ambig_subtrees: -17> 2023-06-14T07:48:56.595+0000 7f14fcb97700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 320109206875743 subtree root 0x1 is not mine in cache (it's -2,-2) -16> 2023-06-14T07:48:56.595+0000 7f14fcb97700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -15> 2023-06-14T07:48:56.595+0000 7f14fcb97700 0 mds.0.journal journal ambig_subtrees: -14> 2023-06-14T07:48:56.598+0000 7f14fcb97700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 320109207491102 subtree root 0x1 is not mine in cache (it's -2,-2) -13> 2023-06-14T07:48:56.598+0000 7f14fcb97700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -12> 2023-06-14T07:48:56.598+0000 7f14fcb97700 0 mds.0.journal journal ambig_subtrees: -11> 2023-06-14T07:48:56.601+0000 7f14fcb97700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 320109208105616 subtree root 0x1 is not mine in cache (it's -2,-2) -10> 2023-06-14T07:48:56.601+0000 7f14fcb97700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -9> 2023-06-14T07:48:56.601+0000 7f14fcb97700 0 mds.0.journal journal ambig_subtrees: -8> 2023-06-14T07:48:56.638+0000 7f1507bad700 10 monclient: get_auth_request con 0x556935cb3400 auth_method 0 -7> 2023-06-14T07:48:56.677+0000 7f15073ac700 10 monclient: get_auth_request con 0x556935c92c00 auth_method 0 -6> 2023-06-14T07:48:56.687+0000 7f1506bab700 10 monclient: get_auth_request con 0x5569359f6400 auth_method 0 -5> 2023-06-14T07:48:56.967+0000 7f1507bad700 10 monclient: get_auth_request con 0x556935cb2800 auth_method 0 -4> 2023-06-14T07:48:57.008+0000 7f15073ac700 10 monclient: get_auth_request con 0x556935cb8400 auth_method 0 -3> 2023-06-14T07:48:57.159+0000 7f1506bab700 10 monclient: get_auth_request con 0x556935cb3800 auth_method 0 -2> 2023-06-14T07:48:57.169+0000 7f1507bad700 10 monclient: get_auth_request con 0x556935cb9000 auth_method 0 -1> 2023-06-14T07:48:57.197+0000 7f15073ac700 10 monclient: get_auth_request con 0x5569359f7400 auth_method 0 0> 2023-06-14T07:48:57.317+0000 7f14fcb97700 -1 *** Caught signal (Segmentation fault) ** in thread 7f14fcb97700 thread_name:md_log_replay

3 2

User + Dev Monthly Meeting Minutes 2023-06-15
by Laura Flores 15 Jun '23

15 Jun '23

Topics: Reef rc is out! https://www.spinics.net/lists/ceph-users/msg77659.html Minutes: [Andras Pataki] Wants to test out Reef as a fresh deploy and as an upgrade; current hardware issues but will update when ready. [Andras Pataki] Is concerned about some performance issues in pacific - Would recommend talking to Mark Nelson about your use case and how it would be affected by any pacific performance issues [Andras Pataki] What are the major changes in Reef? - Major changes in Reef include a RocksDB version upgrade and new RocksDB default tunings [Bryan Stillwell] Was there an improvement to the balancer? - Yes, the read balancer was added as an offline tool available in Reef. [Bryan Stillwell] Were there any improvements to MDS? - Would recommend asking Patrick Donnelly or Greg Farnum -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

RGW bucket not getting create for recreated user via ceph Dashboard
by Maaz Azmi 15 Jun '23

15 Jun '23

Dear Team, I am facing one issue, which could be a possible bug, but not able to find any solution. We are unable to create a bucket on ceph from ceph-dashboard. Bucket is getting created with a fresh/different name but once I am trying with a deleted bucket name then it is not getting created. I have tested on (Octopus and Pacific version). Steps to reproduce. a. Create user RGW user from ceph Dashboard. (user1) b. Create a bucket with the bucket owner as user1. (bucket1) c. Delete the bucket d Delete user. e. Create a user again with the same name (user1) f. Create a new bucket with the bucket owner as user1. (Bucket2) I get below error message: RGW REST API failed request with status code 403 (b'{"Code":"InvalidAccessKeyId","RequestId":"tx00000457ff5169168a9e3-00648afcd0' b'-fdd1c2-dev","HostId":"fdd1c2-dev-india"}') So it seems the ceph dashboard has stored the user (keys) (created in step a.) somewhere and when the same user is created it is not getting updated at ceph dashboard cache. For a temporary fix I have restarted the ceph mgr and then I was able to create a bucket with the same user name. Regards, Maaz Regards, Maaz

1 0

Bottleneck between loadbalancer and rgws
by Szabo, Istvan (Agoda) 15 Jun '23

15 Jun '23

Hi, I have a dedicated loadbalancer pairs separated on 2x baremetal servers and behind the haproxy balancers I have 3 mon/mgr/rgw nodes. Each rgw node has 2rgw on it so in the cluster altogether 6, (now I just added one more so currently 9). Today I see pretty high GET latency in the cluster (3-4s) and seems like the limitations are the gateways: https://i.ibb.co/ypXFL34/1.png In this netstat seems like maxed out the established connections around 2-3k. When I've added one more gateway it increased. Seems like the gateway node or the gateway instance has some limitation. What is the value which is around 1000,I haven't really found it and affect GET and limit the connections on linux? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3

[rgw] Multi-zone per storage cluster
by Yixin Jin 14 Jun '23

14 Jun '23

Hi folks, For multi-site environment, it seems that the common practice is to have one rgw zone backed by one storage cluster. However, after gaining some experience with such a setup, it appears to me that it is possible to have multiple zones with the same storage cluster. The first evidence is that when configuring the multi-site setup, the default zone "default" is still there and it needs a separate step to remove. It seems to me that there are 2 zones co-existing. The second evidence is that I can do "radosgw-admin zone create" multiple times at the same cluster with different zone names and make them all join the same zonegroup, although only one of them is the default. So, is it true that by design, we can have multiple zones per cluster? If so, is it a by-product of the design or a feature? Are there any such use cases in production? Thanks a lot Yixin

1 0

Tuning RGW rgw_object_stripe_size and rgw_max_chunk_size
by David Oganezov 14 Jun '23

14 Jun '23

Hello! In my company we are using an EC 8+4 pool backed by HDDs. Since the default rgw_max_chunk_size and rgw_object_stripe_size parameters are set to 4MiB, each HDD's chunk size is 512KiB, which is not ideal. I was wondering if anyone tried playing with this parameters, as increasing them to 32MiB for example, can theoretically double the throughput of the cluster, as each HDD's chunk will be at 4MiB, which is more suited to HDDs. Thanks!

1 0

Upgrading standard Debian packages
by Ben Thompson 14 Jun '23

14 Jun '23

Hi Everyone, I am new to ceph and looking at how to upgrade a Debian 10 cluster to Debian 11. Each node is running the stock Debian (Luminous) packages and the upgrade to Bullseye (Debian 11) will pull in Nautilus. According to the documentation here https://docs.ceph.com/en/latest/releases/nautilus/#upgrading-from-mimic-or-… the procedure is to upgrade each daemon one by one (MON -> MGR -> OSD). However, it seems that the Debian packages are all interdependent on ceph-base which means that upgrading one at a time is not possible. I would be grateful for any advice. -- Ben Thompson

1 0

Updating the Grafana SSL certificate in Quincy
by Thorne Lawler 13 Jun '23

13 Jun '23

Hi everyone! I have a containerised (cephadm built) 17.2.6 cluster where I have installed a custom commercial SSL certificate under dashboard. Before I upgraded from 17.2 to 17.2.6, I successfully installed the custom SSL cert everywhere, including grafana, but since the upgrade I am finding that I can't update the certificate for grafana. Have tried many many commands like the following: ceph config-key set mgr/cephadm/grafana_crt -i /etc/pki/tls/certs/_.quick.net.au_2024.pem ceph orch reconfig grafana ceph dashboard set-grafana-frontend-api-url https://san.quick.net.au:3000 restorecon /etc/pki/tls/certs/_.quick.net.au_2024.pem ceph orch reconfig grafana ceph dashboard set-grafana-frontend-api-url https://san.quick.net.au:3000 ceph dashboard set-grafana-frontend-url https://san.quick.net.au:3000 ceph dashboard grafana ceph dashboard grafana dashboards update ceph orch reconfig grafana ceph config-key set mgr/cephadm/grafana_crt -i /etc/pki/tls/certs/_.quick.net.au_2024.pem ceph orch redeploy grafana ceph config set mgr mgr/dashboard/GRAFANA_API_URL https://san.quick.net.au:3000 ...but to no avail. The grafana frames within dashboard continue to use the self-signed key. Have the commands for updating this changed between 17.2.0 and 17.2.6? Thank you. -- Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

2 2

How to release the invalid tcp connection under radosgw?
by Louis Koo 13 Jun '23

13 Jun '23

connections: [root@et-uos-warm02 deeproute]# netstat -anltp | grep rados | grep 10.x.x.x:7480 | grep ESTAB | grep 10.12 | wc -l 6650 The prints: tcp 0 0 10.x.x.x:7480 10.x.x.12:40210 ESTABLISHED 76749/radosgw tcp 0 0 10.x.x.x:7480 10.x.x.12:33218 ESTABLISHED 76749/radosgw tcp 0 0 10.x.x.x:7480 10.x.x.12:33546 ESTABLISHED 76749/radosgw tcp 0 0 10.x.x.x:7480 10.x.x.12:50024 ESTABLISHED 76749/radosgw .... but client ip 10.x.x.12 is unreachable(because the node was shutdown), the status of the tcp connections is always "ESTABLISHED", how to fix it?

2 3

log file timestamp precision
by alienfirstzen＠yahoo.com.tw 13 Jun '23

13 Jun '23

Hi,In my ceph log file, the timestamp shows only to millisecond, like below. How to make it show microsecond and nanosecond? By config file, or need to modify source code and where? 2023-06-10T09:38:50.549+0800 7f639d009040 15 client.1500594.objecter _session_op_assign 8 1... My ceph version is ceph:v16 docker image Many thanks!

1 0