- ceph-users - lists.ceph.io

Important Community Updates [Ceph Developer Summit, Cephalocon]
by Noah Lehman 12 Aug '24

12 Aug '24

Hi Ceph community, I have two important updates to share with you about the Ceph Developer Summit and Cephalocon 20204. *Ceph Developer Summit* The Ceph Developer Summit has been extended until August 20th. Find everything you need to know about the event, including the program and other important details, here <https://ceph.io/en/community/events/2024/ceph-developer-summit-tentacle/>. *Cephalocon 2024* The CFP deadline to submit a speaking proposal has been extended until August 18th. If you're interested in participating as a speaker at the event, you can send in a proposal here <https://ceph.io/en/community/events/2024/cephalocon-2024/>. Thanks everyone! Noah

1 0

Search for a professional service to audit a CephFS infrastructure
by Fabien Sirjean 12 Aug '24

12 Aug '24

Hello, In a professional context, I'm looking for someone with strong CephFS expertise to help us audit our infrastructure. We prefer an on-site audit, but are open to working remotely, and can provide any documentation or information required. Please note that we are not currently in a blocking situation (the storage is functional and meets requirements), but we would like to anticipate an increase in workload and identify bottlenecks and areas for improvement. Our infrastructure, located in Grenoble (France), consists of 12 servers in three server rooms, each server containing 60 HDD of 18.2 TiB, for a total of ~12 PiB raw and 4 PiB net (replicated size 3). We currently store 1.72 PiB (net) of data, in a single CephFS filesystem with ~300 clients (linux kernel). A specific pool is dedicated to MDS, backed with NVMe OSDs. Network is 2 x 25Gbps per server and 2 x 100Gbps between the three server rooms, with jumbo frames for the replication network. This CephFS storage houses the scientific data produced by some fifty fundamental research instruments, as well as the analysis of these data. Our questions mainly concern the configuration of the MDS (currently 6 active MDS + 6 standby-replay), such as : - Best number of MDS for our workload - Pinning MDS to specific parts of the filesystem ? - MDS cache size - CephFS clients performances - ... If you'd like to keep in touch (especially commercially), don't hesitate to contact me at my work address: sirjean(a)ill.fr Cheers, -- Fabien Sirjean Head of IT Infrastructures (DPT/SI/INFRA) Institut Laue Langevin (ILL) +33 (0)4 76 20 76 46 / +33 (0)6 62 47 52 80

1 0

Re: Please guide us inidentifyingthecauseofthedata miss in EC pool
by Best Regards 12 Aug '24

12 Aug '24

Hi, Frédéric Thanks for your advice and suggestions. The failure to identify the root cause of the data loss has a certain impact on subsequent improvement measures.min_size does indeed need to be changed to K+1. We will also reevaluate the disaster recovery situation to better handle extreme scenarios. Thank you again. Best regards. Best Regards wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/9 16:33 To:"Best Regards"< wu_chulin(a)qq.com >; Cc recipient:"ceph-users"< ceph-users(a)ceph.io >; Subject:Re: Re:Re: Re:[ceph-users] Re: Please guide us inidentifyingthecauseofthedata miss in EC pool Hi Chulin, When it comes to data consistency, it's generally admitted that Ceph is an undefeated master. Considering the very few (~100) rados objects that were completely lost (data and metadata) and the fact that you're using colocated HDD OSDs with volatile disk buffers caching rocksdb metadata and Bluestore data and metadata, I doubt that volatile disk buffers weren't involved in the data loss, whatever the logs say or don't say about which of the 6 over 9 OSDs were in the acting set at the moment of the power outage. Unless you're ok with facing data loss again, I'd advise you fix the initial design flaws if you can. Like stop using non-persistent cache / buffers along the IO path, raise mon_size to k+1 and reconsider data placement in regards to risks of network partitioning, power outage, fire. Also, considering the ceph status, make sure you don't run out of disk space. Best regards, Frédéric. De : Best Regards <wu_chulin(a)qq.com> Envoyé : jeudi 8 août 2024 11:32 À : Frédéric Nass Cc: ceph-users Objet : Re:Re: Re:[ceph-users] Re: Please guide us inidentifying thecauseofthedata miss in EC pool Hi，Frédéric Nass Sorry, I may not have expressed it clearly before. The epoch and OSD up/down timeline was extracted and merged from the 9 OSD logs. I analyzed the PG (9.11b6) peering process. OSD 494, 1169, 1057 fully recorded the down/up of other OSDs. I also checked the logs of the other 6 OSDs. The role conversion during peering was expected and no abnormalities were found. I also checked the status of the monitor. One of the 5 monitors lost power and was powered on after about 40 minutes. The log showed that its rank value was relatively large and it did not become the leader. Let's talk about the fault domain. The fault domain we set is the host level, but in fact all hosts are distributed in 2 buildings, but the original designer did not consider the fault level of the building. In this case, the OSD may have a brain split, but from the log, it does not happen. Best regards. Best Regards wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 15:40 To:"Best Regards"< wu_chulin(a)qq.com >; Cc recipient:"ceph-users"< ceph-users(a)ceph.io >; Subject:Re: Re:[ceph-users] Re: Please guide us inidentifying thecauseofthedata miss in EC pool ceph osd pause is a lot of constraints from an operational perspective. :-) host uptime and service running time is a thing. But it doesn't mean that these 3 OSDs were in the acting set when the power outage occured. Since OSDs 494, 1169 and 1057 did not crash, I assume they're in the same failure domain. Is that right? Being isolated along with their local MON(s) from other MONs and other 6 OSDs, there's a fair chance that any of the 6 other OSDs in other failure domains took the lead, sent 5 chunks around and acknowledged the write to RGW client. Then all of them crashed. Your thoughts? Frédéric. De : Best Regards <wu_chulin(a)qq.com> Envoyé : jeudi 8 août 2024 09:16 À : Frédéric Nass Cc: ceph-users Objet : Re:[ceph-users] Re: Please guide us inidentifying thecause ofthedata miss in EC pool Hi, Frédéric Nass Yes. I checked the host running time where the OSD is located and the OSD service running time.These were stopped when I executed `ceph-object-tool`. They were running before. Because we need to maintain the hardware frequently (the hardware is quite old), min_size is set to the lowest value. When a failure occurs, we set the read and write pause flag. During the failure, there is no PUT action on the S3 keys to which these objects belong. Best Regards Best Regards wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 14:40 To:"Best Regards"< wu_chulin(a)qq.com >; Cc recipient:"ceph-users"< ceph-users(a)ceph.io >; Subject:[ceph-users] Re: Please guide us inidentifying thecause ofthedata miss in EC pool Hi Chulin, Are you 100% sure that 494, 1169 and 1057 (that did not restart) were in the acting set at the exact moment the power outage occured?  I'm asking because min_size 6 would have allowed the data to be written to eventually 6 crashing OSDs. Bests, Frédéric. ________________________________ De : Best Regards Envoyé : jeudi 8 août 2024 08:10 À : Frédéric Nass Cc: ceph-users Objet : Re:Re: Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us inidentifying thecause ofthedata miss in EC pool Hi， Frédéric Nass Thank you for your continued attention and guidance. Let's analyze and verify this issue from different perspectives. The reason why we did not stop the investigation is that we tried to find other ways to avoid the losses caused by this sudden failure. Turning off the disk cache is the last option, of course, this operation will only be carried out after finding definite evidence. I also have a question that among the 9 OSDs, some have not been restarted. In theory, these OSDs should retain the object info(metadata,pg_log,etc.), even if the object cannot be recovered. I sorted out the OSD booting log where the object should be located and the PG peering process: OSD 494/1169/1057 has been in the running state, and osd.494 was the primary of the acting_set during the failure. However, no record of the object was found using `ceph-object-tool --op list or --op log` in, so the loss of data due to disk cache loss does not seem to be a complete explanation (perhaps there is some processing logic that we have not paid attention to). Best Regards, Woo wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 4:01 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us inidentifying thecause ofthedata miss in EC pool Hey Chulin, Looks clearer now. Non-persistent cache for KV metadata and Bluestore metadata certainly explains how data was lost without the cluster even noticing. What's unexpected is data staying for so long in the disks buffers and not being written to persistent sectors at all. Anyways, thank you for sharing your use case and investigation. It was nice chatting with you. If you can, share this in the ceph-user list. It will for sure benefit everyone in the community. Best regards, Frédéric. PS : Note that using min_size >= k + 1 on EC pools is recommended (so as min_size >= 2 on rep X3 pools) because you don't want to write data without any parity chunks. De : wu_chulin(a)qq.com Envoyé : mercredi 7 août 2024 11:30 À : Frédéric Nass Objet : Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us in identifying thecause ofthedata miss in EC pool Hi, Yes, after the file -> object -> PG -> OSD correspondence is found, the object record can be found on the specified OSD using the command `ceph-objectore-tool --op list ` The pool min_size is 6 The business department reported more than 30, but we proactively screened out more than 100. The upload time of the lost files was mainly distributed about 3 hours before the failure, and these files were successfully downloaded after being uploaded (RGW log). One OSD corresponds to one disk, and no separate space is allocated for WAL/DB. The HDD cache is the default (SATA is enabled by default), and the hard disk cache has not been forcibly turned off due to performance issues. The loss of OSD data due to the loss of hard disk cache was our initial inference, and the initial explanation provided to the business department was the same. When the cluster was restored, ceph reported 12 unfound objects, which is acceptable. After all, most devices were powered off abnormally, and it is difficult to ensure the integrity of all data. Up to now, our team have not located how the data was lost. In the past, when the hard disk hardware was damaged, either the OSD could not start because of damaged key data, or some objects were read incorrectly after the OSD started, which could be repaired. Now deep-scrub cannot find the problem, which may be related to the loss (or deletion) of object metadata. After all, deep-scrub needs the object list of the current PG. If those 9 OSDs do not have the object metadata information, deep-scrub does not know the existence of this object. wu_chulin(a)qq.com wu_chulin(a)qq.com Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/6 20:40 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: Re:Re: [ceph-users] Please guide us in identifying thecause ofthedata miss in EC pool That's interesting. Have you tried to correlate any existing retrievable object to PG id and OSD mapping in order to verify the presence of each of these object's shards using ceph-objectore-tool on each one of its acting OSDs, for a previously and successfully written S3 object. This would help verify that the command you've run trying to find the missing shard was good. Also what min_size is this pool using?    How many S3 objects like these were reported missing by the business unit? Have you or they made an inventory of unretrievable / missing objects? Are WAL / RocksDB collocated on HDDs only OSDs or are these OSDs using SSDs for WAL / RocksDB? Did you disable HDD buffers (also known as disk cache) as... HDD buffers are non-persistent. I know nothing about the intensity of your workloads but if your looking for a few tens or a few hundreds of unwritten s3 objects, there might be some situation with non-persistent cache (like volatile disk buffers) were ceph would consider the data to be written when it was actually not at the moment of the power outage, due to the use of non-persistent disk buffers. Especially if you kept writing data with less shards (min_size) than k+1 (no parity at all). That sound like a possibility. Also what I'm thinking right is... If you can identify which shard over 9 is wrong, then you may use ceph-objectore-tool or ceph-kvstore-tool to destroy this particular shard, then deep-scrub the PG so to detect the inexistent shard and have it rebuilt. Never tried this myself though. Best regards, Frédéric. De : wu_chulin(a)qq.com Envoyé : mardi 6 août 2024 12:15 À : Frédéric Nass Objet : Re:Re: Re:Re: [ceph-users] Please guide us in identifying the cause ofthedata miss in EC pool Hi,  Thank you for your attention to this matter. 1. Manually executing deep-scrub PG did not report any errors. I checked the OSD logs and did not see any errors detected or fixed by the OSD. Ceph health was also normal, and the OS where the OSD was located did not report any IO type errors. 2. At first, I also suspected whether the objects were distributed on these OSDs before the failure. I used ceph osd getmap 3. Our Ceph version is 13.2.10, which does not have pg autoscaler; lifecycle policies are not set. 4. We have 90+ hosts, most of which are Dell R720xd, most of the hard disks are 3.5 inches/5400 rpm/10TB Western Digital/, and most of the controllers are PERC H330 Mini; this is the current cluster status: cluster: id: f990db28-9604-4d49-9733-b17155887e3b health: HEALTH_OK services:   mon: 5 daemons, quorum cz-ceph-01,cz-ceph-02,cz-ceph-03,cz-ceph-07,cz-ceph-13   mgr: cz-ceph-01(active), standbys: cz-ceph-03, cz-ceph-02   osd: 1172 osds: 1172 up, 1172 in    rgw: 9 daemons active  data:    pools: 16 pools, 25752 pgs    objects: 2.23 G objects, 6.1 PiB    usage: 9.4 PiB used, 2.5 PiB / 12 PiB avail    pgs:      25686 active+clean                 64 active+clean+scrubbing+deep+repair                 2 active+clean+scrubbing+deep Best regards. Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/6 15:34 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: [ceph-users] Please guide us in identifying the cause ofthedata miss in EC pool Hi, Did the deep-scrub report any errors? (any inconsistencies should show errors after deep-scrubbing the PG.)   We're these errors fixed by the PG repair? Is is possible that you looked for the wrong PG or OSD when trying to list these objects with ceph-objectstore-tool? Were the PG autoscaler running at that time? Are you using S3 lifecycle policies that could have move this object to another placement pool and so another PG? Can you give details about this cluster? Hardware, disks, controller, etc. Cheers, Frédéric. De : wu_chulin(a)qq.com Envoyé : lundi 5 août 2024 10:09 À : Frédéric Nass Objet : Re:Re: [ceph-users] Please guide us in identifying the cause of thedata miss in EC pool Hi， Thank you for your reply. I apologize for the omission in the previous email. Please disregard the previous email and refer to this one instead.  After the failure, we executed the repair and deep-scrub command on some of the PGs that lost data, and the status was active+clean after completion, but the object still could not be retrieved. Our erasure code parameters are k=6, m=3. Theoretically, the data of three OSDs lost due to power failure should be recoverable. However, we stopped nine OSDs and exported the object list, but could not find the lost object information. What puzzled us was that some OSDs were not powered off and were still running, but their object lists did not have the information too. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/3 15:11 To:"wu_chulin"< wu_chulin(a)qq.com >;"ceph-users"< ceph-users(a)ceph.io >; Subject:Re: [ceph-users] Please guide us in identifying the cause of thedata miss in EC pool Hi, First thing that comes to mind when it comes to data unavailability or inconsistencies after a power outage is that some dirty data may have been lost along the IO path before reaching persistent storage. This can happen with non enterprise grade SSDs using non-persistent cache or with HDDs disk buffer if left enabled for example. With that said, have you tried to deep-scrub the PG from which you can't retrieve data? What's the status of this PG now? Did it recover? Regards, Frédéric. De : wu_chulin(a)qq.com Envoyé : mercredi 31 juillet 2024 05:49 À : ceph-users Objet : [ceph-users] Please guide us in identifying the cause of the data miss in EC pool Dear Ceph team:&nbsp; &nbsp; &nbsp;On July 13th at 4:55 AM, our Ceph cluster experienced a significant power outage in the data center, causing a large number of OSDs to power off and restart (total: 1172, down: 821). Approximately two hours later, all OSDs successfully started, and the cluster resumed its services. However, around 6 PM, the business department reported that some files, which had been successfully written (via the RGW service), were failing to download, and the number of such files was quite significant. Consequently, we began a series of investigations: 1. The incident occurred at 04:55. At 05:01, we executed noout, nobackfill, and norecover. At 06:22, we executed `ceph osd pause`. By 07:23, all OSDs were UP&amp;IN, and subsequently, we executed `ceph osd unpause`. 2. We randomly selected a problematic file and attempted to download it via the S3 API. The RGW returned "No such key". 3. The RGW logs showed op status=-2, http status=200. We also checked the upload logs, which indicated 2024-07-13 04:19:20.052, op status=0, http_status=200. 4. We set debug_rgw=20 and attempted to download the file again. It was found that a 4M chunk(this file is 64M) failed to get. 5. Using rados get for this chunk returned: "No such file or directory". 6. Setting debug_osd=20, we observed get_object_context: obc NOT found in cache. 7. Setting debug_bluestore=20, we saw get_onode oid xxx, key xxx != '0xfffffffffffffffeffffffffffffffff'o'. 8. We stopped the primary OSD and tried to get the file again, but the result was the same. The object’s corresponding PG state was active+recovery_wait+degraded. 9. Using ceph-objectstore-tool --op list &amp;&amp; --op log, we could not find the object information. The ceph-kvstore-tool rocksdb command also did not reveal anything new. 10. If an OSD had lost data, the PG state should have been unfound or inconsistency. 11. We started reanalyzing the startup logs of the OSDs related to the PG. The pool was set to erasure-code 6-3 with 9 OSDs. Six of these OSDs had restarted, and after peering, the PG state became ACTIVE. 12. We divided the lost files, and the upload time was before the failure occurred. The earliest upload time was around 1 am, and the successful upload records could be found in the RGW log 13. We have submitted an issue on the Ceph issue tracker:&nbsp;https://tracker.ceph.com/issues/66942, it includes the original logs needed for troubleshooting. However, four days have passed without any response. In desperation, we are sending this email, hoping that someone from the Ceph team can guide us as soon as possible. We are currently in a difficult situation and hope you can provide guidance. Thank you. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Any way to put the rate limit on rbd flatten operation?
by Henry lol 12 Aug '24

12 Aug '24

Hello, AFAIK, massive rx/tx occurs on the client side for the flatten operation. so, I want to control the network rate limit or predict the network bandwidth it will consume. Is there any way to do that?

2 1

[Cephalocon 2024] CFP Closes TOMORROW
by Patrick Donnelly 10 Aug '24

10 Aug '24

Cephalocon is the premier yearly event that brings together the global community of operators, developers, and researchers for Ceph, the open source distributed storage system. The CFP closes tomorrow, 2024-08-11. Please submit your talk proposals: https://ceph.io/en/community/events/2024/cephalocon-2024/ Regards, -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: Please guide us inidentifying thecauseofthedata miss in EC pool
by Best Regards 09 Aug '24

09 Aug '24

Hi，Frédéric Nass Sorry, I may not have expressed it clearly before. The epoch and OSD up/down timeline was extracted and merged from the 9 OSD logs. I analyzed the PG (9.11b6) peering process. OSD 494, 1169, 1057 fully recorded the down/up of other OSDs. I also checked the logs of the other 6 OSDs. The role conversion during peering was expected and no abnormalities were found. I also checked the status of the monitor. One of the 5 monitors lost power and was powered on after about 40 minutes. The log showed that its rank value was relatively large and it did not become the leader. Let's talk about the fault domain. The fault domain we set is the host level, but in fact all hosts are distributed in 2 buildings, but the original designer did not consider the fault level of the building. In this case, the OSD may have a brain split, but from the log, it does not happen. Best regards. Best Regards wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 15:40 To:"Best Regards"< wu_chulin(a)qq.com >; Cc recipient:"ceph-users"< ceph-users(a)ceph.io >; Subject:Re: Re:[ceph-users] Re: Please guide us inidentifying thecauseofthedata miss in EC pool ceph osd pause is a lot of constraints from an operational perspective. :-) host uptime and service running time is a thing. But it doesn't mean that these 3 OSDs were in the acting set when the power outage occured. Since OSDs 494, 1169 and 1057 did not crash, I assume they're in the same failure domain. Is that right? Being isolated along with their local MON(s) from other MONs and other 6 OSDs, there's a fair chance that any of the 6 other OSDs in other failure domains took the lead, sent 5 chunks around and acknowledged the write to RGW client. Then all of them crashed. Your thoughts? Frédéric. De : Best Regards <wu_chulin(a)qq.com> Envoyé : jeudi 8 août 2024 09:16 À : Frédéric Nass Cc: ceph-users Objet : Re:[ceph-users] Re: Please guide us inidentifying thecause ofthedata miss in EC pool Hi, Frédéric Nass Yes. I checked the host running time where the OSD is located and the OSD service running time.These were stopped when I executed `ceph-object-tool`. They were running before. Because we need to maintain the hardware frequently (the hardware is quite old), min_size is set to the lowest value. When a failure occurs, we set the read and write pause flag. During the failure, there is no PUT action on the S3 keys to which these objects belong. Best Regards Best Regards wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 14:40 To:"Best Regards"< wu_chulin(a)qq.com >; Cc recipient:"ceph-users"< ceph-users(a)ceph.io >; Subject:[ceph-users] Re: Please guide us inidentifying thecause ofthedata miss in EC pool Hi Chulin, Are you 100% sure that 494, 1169 and 1057 (that did not restart) were in the acting set at the exact moment the power outage occured?  I'm asking because min_size 6 would have allowed the data to be written to eventually 6 crashing OSDs. Bests, Frédéric. ________________________________ De : Best Regards Envoyé : jeudi 8 août 2024 08:10 À : Frédéric Nass Cc: ceph-users Objet : Re:Re: Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us inidentifying thecause ofthedata miss in EC pool Hi， Frédéric Nass Thank you for your continued attention and guidance. Let's analyze and verify this issue from different perspectives. The reason why we did not stop the investigation is that we tried to find other ways to avoid the losses caused by this sudden failure. Turning off the disk cache is the last option, of course, this operation will only be carried out after finding definite evidence. I also have a question that among the 9 OSDs, some have not been restarted. In theory, these OSDs should retain the object info(metadata,pg_log,etc.), even if the object cannot be recovered. I sorted out the OSD booting log where the object should be located and the PG peering process: OSD 494/1169/1057 has been in the running state, and osd.494 was the primary of the acting_set during the failure. However, no record of the object was found using `ceph-object-tool --op list or --op log` in, so the loss of data due to disk cache loss does not seem to be a complete explanation (perhaps there is some processing logic that we have not paid attention to). Best Regards, Woo wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 4:01 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us inidentifying thecause ofthedata miss in EC pool Hey Chulin, Looks clearer now. Non-persistent cache for KV metadata and Bluestore metadata certainly explains how data was lost without the cluster even noticing. What's unexpected is data staying for so long in the disks buffers and not being written to persistent sectors at all. Anyways, thank you for sharing your use case and investigation. It was nice chatting with you. If you can, share this in the ceph-user list. It will for sure benefit everyone in the community. Best regards, Frédéric. PS : Note that using min_size >= k + 1 on EC pools is recommended (so as min_size >= 2 on rep X3 pools) because you don't want to write data without any parity chunks. De : wu_chulin(a)qq.com Envoyé : mercredi 7 août 2024 11:30 À : Frédéric Nass Objet : Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us in identifying thecause ofthedata miss in EC pool Hi, Yes, after the file -> object -> PG -> OSD correspondence is found, the object record can be found on the specified OSD using the command `ceph-objectore-tool --op list ` The pool min_size is 6 The business department reported more than 30, but we proactively screened out more than 100. The upload time of the lost files was mainly distributed about 3 hours before the failure, and these files were successfully downloaded after being uploaded (RGW log). One OSD corresponds to one disk, and no separate space is allocated for WAL/DB. The HDD cache is the default (SATA is enabled by default), and the hard disk cache has not been forcibly turned off due to performance issues. The loss of OSD data due to the loss of hard disk cache was our initial inference, and the initial explanation provided to the business department was the same. When the cluster was restored, ceph reported 12 unfound objects, which is acceptable. After all, most devices were powered off abnormally, and it is difficult to ensure the integrity of all data. Up to now, our team have not located how the data was lost. In the past, when the hard disk hardware was damaged, either the OSD could not start because of damaged key data, or some objects were read incorrectly after the OSD started, which could be repaired. Now deep-scrub cannot find the problem, which may be related to the loss (or deletion) of object metadata. After all, deep-scrub needs the object list of the current PG. If those 9 OSDs do not have the object metadata information, deep-scrub does not know the existence of this object. wu_chulin(a)qq.com wu_chulin(a)qq.com Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/6 20:40 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: Re:Re: [ceph-users] Please guide us in identifying thecause ofthedata miss in EC pool That's interesting. Have you tried to correlate any existing retrievable object to PG id and OSD mapping in order to verify the presence of each of these object's shards using ceph-objectore-tool on each one of its acting OSDs, for a previously and successfully written S3 object. This would help verify that the command you've run trying to find the missing shard was good. Also what min_size is this pool using?    How many S3 objects like these were reported missing by the business unit? Have you or they made an inventory of unretrievable / missing objects? Are WAL / RocksDB collocated on HDDs only OSDs or are these OSDs using SSDs for WAL / RocksDB? Did you disable HDD buffers (also known as disk cache) as... HDD buffers are non-persistent. I know nothing about the intensity of your workloads but if your looking for a few tens or a few hundreds of unwritten s3 objects, there might be some situation with non-persistent cache (like volatile disk buffers) were ceph would consider the data to be written when it was actually not at the moment of the power outage, due to the use of non-persistent disk buffers. Especially if you kept writing data with less shards (min_size) than k+1 (no parity at all). That sound like a possibility. Also what I'm thinking right is... If you can identify which shard over 9 is wrong, then you may use ceph-objectore-tool or ceph-kvstore-tool to destroy this particular shard, then deep-scrub the PG so to detect the inexistent shard and have it rebuilt. Never tried this myself though. Best regards, Frédéric. De : wu_chulin(a)qq.com Envoyé : mardi 6 août 2024 12:15 À : Frédéric Nass Objet : Re:Re: Re:Re: [ceph-users] Please guide us in identifying the cause ofthedata miss in EC pool Hi,  Thank you for your attention to this matter. 1. Manually executing deep-scrub PG did not report any errors. I checked the OSD logs and did not see any errors detected or fixed by the OSD. Ceph health was also normal, and the OS where the OSD was located did not report any IO type errors. 2. At first, I also suspected whether the objects were distributed on these OSDs before the failure. I used ceph osd getmap 3. Our Ceph version is 13.2.10, which does not have pg autoscaler; lifecycle policies are not set. 4. We have 90+ hosts, most of which are Dell R720xd, most of the hard disks are 3.5 inches/5400 rpm/10TB Western Digital/, and most of the controllers are PERC H330 Mini; this is the current cluster status: cluster: id: f990db28-9604-4d49-9733-b17155887e3b health: HEALTH_OK services:   mon: 5 daemons, quorum cz-ceph-01,cz-ceph-02,cz-ceph-03,cz-ceph-07,cz-ceph-13   mgr: cz-ceph-01(active), standbys: cz-ceph-03, cz-ceph-02   osd: 1172 osds: 1172 up, 1172 in    rgw: 9 daemons active  data:    pools: 16 pools, 25752 pgs    objects: 2.23 G objects, 6.1 PiB    usage: 9.4 PiB used, 2.5 PiB / 12 PiB avail    pgs:      25686 active+clean                 64 active+clean+scrubbing+deep+repair                 2 active+clean+scrubbing+deep Best regards. Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/6 15:34 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: [ceph-users] Please guide us in identifying the cause ofthedata miss in EC pool Hi, Did the deep-scrub report any errors? (any inconsistencies should show errors after deep-scrubbing the PG.)   We're these errors fixed by the PG repair? Is is possible that you looked for the wrong PG or OSD when trying to list these objects with ceph-objectstore-tool? Were the PG autoscaler running at that time? Are you using S3 lifecycle policies that could have move this object to another placement pool and so another PG? Can you give details about this cluster? Hardware, disks, controller, etc. Cheers, Frédéric. De : wu_chulin(a)qq.com Envoyé : lundi 5 août 2024 10:09 À : Frédéric Nass Objet : Re:Re: [ceph-users] Please guide us in identifying the cause of thedata miss in EC pool Hi， Thank you for your reply. I apologize for the omission in the previous email. Please disregard the previous email and refer to this one instead.  After the failure, we executed the repair and deep-scrub command on some of the PGs that lost data, and the status was active+clean after completion, but the object still could not be retrieved. Our erasure code parameters are k=6, m=3. Theoretically, the data of three OSDs lost due to power failure should be recoverable. However, we stopped nine OSDs and exported the object list, but could not find the lost object information. What puzzled us was that some OSDs were not powered off and were still running, but their object lists did not have the information too. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/3 15:11 To:"wu_chulin"< wu_chulin(a)qq.com >;"ceph-users"< ceph-users(a)ceph.io >; Subject:Re: [ceph-users] Please guide us in identifying the cause of thedata miss in EC pool Hi, First thing that comes to mind when it comes to data unavailability or inconsistencies after a power outage is that some dirty data may have been lost along the IO path before reaching persistent storage. This can happen with non enterprise grade SSDs using non-persistent cache or with HDDs disk buffer if left enabled for example. With that said, have you tried to deep-scrub the PG from which you can't retrieve data? What's the status of this PG now? Did it recover? Regards, Frédéric. De : wu_chulin(a)qq.com Envoyé : mercredi 31 juillet 2024 05:49 À : ceph-users Objet : [ceph-users] Please guide us in identifying the cause of the data miss in EC pool Dear Ceph team:&nbsp; &nbsp; &nbsp;On July 13th at 4:55 AM, our Ceph cluster experienced a significant power outage in the data center, causing a large number of OSDs to power off and restart (total: 1172, down: 821). Approximately two hours later, all OSDs successfully started, and the cluster resumed its services. However, around 6 PM, the business department reported that some files, which had been successfully written (via the RGW service), were failing to download, and the number of such files was quite significant. Consequently, we began a series of investigations: 1. The incident occurred at 04:55. At 05:01, we executed noout, nobackfill, and norecover. At 06:22, we executed `ceph osd pause`. By 07:23, all OSDs were UP&amp;IN, and subsequently, we executed `ceph osd unpause`. 2. We randomly selected a problematic file and attempted to download it via the S3 API. The RGW returned "No such key". 3. The RGW logs showed op status=-2, http status=200. We also checked the upload logs, which indicated 2024-07-13 04:19:20.052, op status=0, http_status=200. 4. We set debug_rgw=20 and attempted to download the file again. It was found that a 4M chunk(this file is 64M) failed to get. 5. Using rados get for this chunk returned: "No such file or directory". 6. Setting debug_osd=20, we observed get_object_context: obc NOT found in cache. 7. Setting debug_bluestore=20, we saw get_onode oid xxx, key xxx != '0xfffffffffffffffeffffffffffffffff'o'. 8. We stopped the primary OSD and tried to get the file again, but the result was the same. The object’s corresponding PG state was active+recovery_wait+degraded. 9. Using ceph-objectstore-tool --op list &amp;&amp; --op log, we could not find the object information. The ceph-kvstore-tool rocksdb command also did not reveal anything new. 10. If an OSD had lost data, the PG state should have been unfound or inconsistency. 11. We started reanalyzing the startup logs of the OSDs related to the PG. The pool was set to erasure-code 6-3 with 9 OSDs. Six of these OSDs had restarted, and after peering, the PG state became ACTIVE. 12. We divided the lost files, and the upload time was before the failure occurred. The earliest upload time was around 1 am, and the successful upload records could be found in the RGW log 13. We have submitted an issue on the Ceph issue tracker:&nbsp;https://tracker.ceph.com/issues/66942, it includes the original logs needed for troubleshooting. However, four days have passed without any response. In desperation, we are sending this email, hoping that someone from the Ceph team can guide us as soon as possible. We are currently in a difficult situation and hope you can provide guidance. Thank you. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

RGW: HEAD ok but GET fails
by Eugen Block 09 Aug '24

09 Aug '24

Hi, I'm trying to help a customer with a RGW question, maybe someone here can help me out. Their S3 application reports errors every now and then, and it is complaining about missing objects. This is what the RGW logs: [08/Aug/2024:08:23:47.540 +0000] "HEAD /hchsarchiv/20240622221326-20540623-aeaa962adadf5bc92050823dd03039197987f9d16f70c793599e361b6a5910c83941a0ceb3c7bfccb0a8ecbae212c701958d8a316b4fb172a54040b26b3a2508 HTTP/1.1" 200 0 - "aws-sdk-dotnet-45/3.5.9.7 aws-sdk-dotnet-core/3.5.3.7 .NET_Runtime/4.0 .NET_Framework/4.0 OS/Microsoft_Windows_NT_10.0.14393.0 ClientSync" - latency=0.003999992s [08/Aug/2024:08:23:47.552 +0000] "GET /hchsarchiv/20240622221326-20540623-aeaa962adadf5bc92050823dd03039197987f9d16f70c793599e361b6a5910c83941a0ceb3c7bfccb0a8ecbae212c701958d8a316b4fb172a54040b26b3a2508 HTTP/1.1" 404 242 - "aws-sdk-dotnet-45/3.5.9.7 aws-sdk-dotnet-core/3.5.3.7 .NET_Runtime/4.0 .NET_Framework/4.0 OS/Microsoft_Windows_NT_10.0.14393.0 ClientSync" bytes=0-2097151 latency=0.003999992s So apparently, it can successfully query the HEAD, but the GET request shows 404. We can confirm that the queried object indeed doesn't exist in the data pool. But the object metadata must have been written successfully, apparently. Unfortunately, we don't have enough logs to find the corresponding PUT request, they just increased the retention days for logrotate to be able to inspect when it happens the next time. But my question is, should they see some metadata in the listomapkeys/listomapvals output in the index pool? The docs [0] state this about Index Transactions: > Because the head objects are stored in different rados objects than > the bucket indices, we can’t update both atomically with a single > rados operation. In order to satisfy the Consistency Guarantee for > listing operations, we have to coordinate these two object writes > using a three-step bucket index transaction: > > 1. Prepare a transaction on its bucket index object. > 2. Write or delete the head object. > 3. Commit the transaction on the bucket index object (or cancel the > transaction if step 2 fails). > > Object writes and deletes may race with each other, so a given > object may have more than one prepared transaction at a time. RGW > considers an object entry to be ‘pending’ if there are any > outstanding transactions, or ‘completed’ otherwise. Could this be such a race condition which "just happens" from time to time? Or can this somehow be prevented from happening? Because right now the clenaup process is a bit complicated application-wise. I'm not the most experienced RGW user, so I'd be grateful for any pointers here. Thanks! Eugen [0] https://docs.ceph.com/en/reef/dev/radosgw/bucket_index/#index-transaction

2 2

Possible regression? Kernel cephfs >= 6.10 cpu hangup
by caskd 09 Aug '24

09 Aug '24

Hello fellow ceph users, i've noticed that since i've upgraded to kernel version 6.10 i've been occasionally having system lockups with dmesg showing cephfs in the kernel trace. Due to the circumstances of the lockups, i've so far been unable to store these trace logs, but i will attempt to do so in a more isolated and controlled environment. The symptom seems to occur during file syncs (flushing), more specifically it has happened 2 times while saving a file from vim (create new file, flush to file, overwrite old file). Is anyone aware of changes that may have caused such a problem before i start digging deeper? Have a nice day -- Alex D. RedXen System & Infrastructure Administration https://redxen.eu/

1 1

(belated) CLT notes
by Gregory Farnum 08 Aug '24

08 Aug '24

Hi folks, the CLT met on Monday August 5. We discussed a few topics: * The mailing lists are a problem to moderate right now with a huge increase in spam. We have two problems: 1) the moderation system's web front-end apparently isn't operational. That's getting fixed. 2) The moderation is a big load regardless. Casey asked if we want to move to the Linux Foundation-managed list system; Josh will take that to the board since it involves some cost. * Squid Release. 19.1.1 testing is making good progress. * (Likely) Final quincy release 17.2.8 is next up in the queue. There isn't a lot targeted for it right now, so please apply the explicit 17.2.8 milestone or set needs-qa on quincy-labeled PRs if you want Yuri to help move them forward. * The big topic is centos8 stream, in two ways: 1) We dropped building packages for it in the middle of the Reef stable release stream because our build system can't handle the way they closed the release. (Packages are still available at archival URLs, but we aren't set up to use those and probably can't access them the same way we normally do.) This has understandably upset some people. Can we fix it? 2) We had many developers spend a lot of time unexpectedly updating our QA systems when 8.stream disappeared, and it's still an ongoing problem for things like upgrade testing of backports. Is Centos stream a reliable base for our container images, package building, and test infrastructure? Should we move elsewhere? We need more information and a wider discussion, so this was also a topic at yesterday's CDM. We will continue to discuss and coordinate with other interest groups and keep you informed. -Greg

1 0

Cephadm: unable to copy ceph.conf.new
by Magnus Larsen 08 Aug '24

08 Aug '24

Hi Ceph-users! Ceph version: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Using cephadm to orchestrate the Ceph cluster I’m running into https://tracker.ceph.com/issues/59189, which is fixed in next version—quincy 17.2.7—via https://github.com/ceph/ceph/pull/50906 But I am unable to upgrade to the fixed version because of that bug When I try to upgrade (using “ceph orch upgrade start –image internal_mirror/ceph:v17.2.7”), we see the same error message: executing _write_files((['dkcphhpcadmin01', 'dkcphhpcmgt028', 'dkcphhpcmgt029', 'dkcphhpcmgt031', 'dkcphhpcosd033', 'dkcphhpcosd034', 'dkcphhpcosd035', 'dkcphhpcosd036', 'dkcphhpcosd037', 'dkcphhpcosd038', 'dkcphhpcosd039', 'dkcphhpcosd040', 'dkcphhpcosd041', 'dkcphhpcosd042', 'dkcphhpcosd043', 'dkcphhpcosd044'],)) failed. Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 240, in _write_remote_file conn = await self._remote_connection(host, addr) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp await source.run(srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run await self._send_files(path, b'') File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files await self._send_file(srcpath, dstpath, attrs) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file await self._make_cd_request(b'C', attrs, size, srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request self._fs.basename(path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request raise exc asyncssh.sftp.SFTPFailure: scp: /tmp/etc/ceph/ceph.conf.new: Permission denied During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/utils.py", line 79, in do_work return f(*arg) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1088, in _write_files self._write_client_files(client_files, host) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1107, in _write_client_files self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 261, in write_remote_file self.mgr.wait_async(self._write_remote_file( File "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async return self.event_loop.get_result(coro) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result return asyncio.run_coroutine_threadsafe(coro, self._loop).result() File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/ssh.py", line 249, in _write_remote_file logger.exception(msg) orchestrator._interface.OrchestratorError: Unable to write dkcphhpcmgt028:/etc/ceph/ceph.conf: scp: /tmp/etc/ceph/ceph.conf.new: Permission denied We were thinking about removing the keyring from the Ceph orchestrator (https://docs.ceph.com/en/latest/cephadm/operations/#putting-a-keyring-under…) which would then make Ceph not try to copy over a new ceph.conf, alleviating the problem (https://docs.ceph.com/en/latest/cephadm/operations/#client-keyrings-and-con…) but in doing so, Ceph will kindly remove the key from all nodes (https://docs.ceph.com/en/latest/cephadm/operations/#disabling-management-of…) leaving us without the admin keyring. So that doesn’t sound like a path we want to take :S Does anybody know how to get around this issue, so I can get to version where the issue fixed for good? Thanks, Magnus

3 3