- ceph-users - lists.ceph.io

Re: Please guide us inidentifying thecause ofthedata miss in EC pool
by Best Regards 08 Aug '24

08 Aug '24

Hi, Frédéric Nass Yes. I checked the host running time where the OSD is located and the OSD service running time.These were stopped when I executed `ceph-object-tool`. They were running before. Because we need to maintain the hardware frequently (the hardware is quite old), min_size is set to the lowest value. When a failure occurs, we set the read and write pause flag. During the failure, there is no PUT action on the S3 keys to which these objects belong. Best Regards Best Regards wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 14:40 To:"Best Regards"< wu_chulin(a)qq.com >; Cc recipient:"ceph-users"< ceph-users(a)ceph.io >; Subject:[ceph-users] Re: Please guide us inidentifying thecause ofthedata miss in EC pool Hi Chulin, Are you 100% sure that 494, 1169 and 1057 (that did not restart) were in the acting set at the exact moment the power outage occured?  I'm asking because min_size 6 would have allowed the data to be written to eventually 6 crashing OSDs. Bests, Frédéric. ________________________________ De : Best Regards Envoyé : jeudi 8 août 2024 08:10 À : Frédéric Nass Cc: ceph-users Objet : Re:Re: Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us inidentifying thecause ofthedata miss in EC pool Hi， Frédéric Nass Thank you for your continued attention and guidance. Let's analyze and verify this issue from different perspectives. The reason why we did not stop the investigation is that we tried to find other ways to avoid the losses caused by this sudden failure. Turning off the disk cache is the last option, of course, this operation will only be carried out after finding definite evidence. I also have a question that among the 9 OSDs, some have not been restarted. In theory, these OSDs should retain the object info(metadata,pg_log,etc.), even if the object cannot be recovered. I sorted out the OSD booting log where the object should be located and the PG peering process: OSD 494/1169/1057 has been in the running state, and osd.494 was the primary of the acting_set during the failure. However, no record of the object was found using `ceph-object-tool --op list or --op log` in, so the loss of data due to disk cache loss does not seem to be a complete explanation (perhaps there is some processing logic that we have not paid attention to). Best Regards, Woo wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 4:01 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us inidentifying thecause ofthedata miss in EC pool Hey Chulin, Looks clearer now. Non-persistent cache for KV metadata and Bluestore metadata certainly explains how data was lost without the cluster even noticing. What's unexpected is data staying for so long in the disks buffers and not being written to persistent sectors at all. Anyways, thank you for sharing your use case and investigation. It was nice chatting with you. If you can, share this in the ceph-user list. It will for sure benefit everyone in the community. Best regards, Frédéric. PS : Note that using min_size >= k + 1 on EC pools is recommended (so as min_size >= 2 on rep X3 pools) because you don't want to write data without any parity chunks. De : wu_chulin(a)qq.com Envoyé : mercredi 7 août 2024 11:30 À : Frédéric Nass Objet : Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us in identifying thecause ofthedata miss in EC pool Hi, Yes, after the file -> object -> PG -> OSD correspondence is found, the object record can be found on the specified OSD using the command `ceph-objectore-tool --op list ` The pool min_size is 6 The business department reported more than 30, but we proactively screened out more than 100. The upload time of the lost files was mainly distributed about 3 hours before the failure, and these files were successfully downloaded after being uploaded (RGW log). One OSD corresponds to one disk, and no separate space is allocated for WAL/DB. The HDD cache is the default (SATA is enabled by default), and the hard disk cache has not been forcibly turned off due to performance issues. The loss of OSD data due to the loss of hard disk cache was our initial inference, and the initial explanation provided to the business department was the same. When the cluster was restored, ceph reported 12 unfound objects, which is acceptable. After all, most devices were powered off abnormally, and it is difficult to ensure the integrity of all data. Up to now, our team have not located how the data was lost. In the past, when the hard disk hardware was damaged, either the OSD could not start because of damaged key data, or some objects were read incorrectly after the OSD started, which could be repaired. Now deep-scrub cannot find the problem, which may be related to the loss (or deletion) of object metadata. After all, deep-scrub needs the object list of the current PG. If those 9 OSDs do not have the object metadata information, deep-scrub does not know the existence of this object. wu_chulin(a)qq.com wu_chulin(a)qq.com Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/6 20:40 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: Re:Re: [ceph-users] Please guide us in identifying thecause ofthedata miss in EC pool That's interesting. Have you tried to correlate any existing retrievable object to PG id and OSD mapping in order to verify the presence of each of these object's shards using ceph-objectore-tool on each one of its acting OSDs, for a previously and successfully written S3 object. This would help verify that the command you've run trying to find the missing shard was good. Also what min_size is this pool using?    How many S3 objects like these were reported missing by the business unit? Have you or they made an inventory of unretrievable / missing objects? Are WAL / RocksDB collocated on HDDs only OSDs or are these OSDs using SSDs for WAL / RocksDB? Did you disable HDD buffers (also known as disk cache) as... HDD buffers are non-persistent. I know nothing about the intensity of your workloads but if your looking for a few tens or a few hundreds of unwritten s3 objects, there might be some situation with non-persistent cache (like volatile disk buffers) were ceph would consider the data to be written when it was actually not at the moment of the power outage, due to the use of non-persistent disk buffers. Especially if you kept writing data with less shards (min_size) than k+1 (no parity at all). That sound like a possibility. Also what I'm thinking right is... If you can identify which shard over 9 is wrong, then you may use ceph-objectore-tool or ceph-kvstore-tool to destroy this particular shard, then deep-scrub the PG so to detect the inexistent shard and have it rebuilt. Never tried this myself though. Best regards, Frédéric. De : wu_chulin(a)qq.com Envoyé : mardi 6 août 2024 12:15 À : Frédéric Nass Objet : Re:Re: Re:Re: [ceph-users] Please guide us in identifying the cause ofthedata miss in EC pool Hi,  Thank you for your attention to this matter. 1. Manually executing deep-scrub PG did not report any errors. I checked the OSD logs and did not see any errors detected or fixed by the OSD. Ceph health was also normal, and the OS where the OSD was located did not report any IO type errors. 2. At first, I also suspected whether the objects were distributed on these OSDs before the failure. I used ceph osd getmap 3. Our Ceph version is 13.2.10, which does not have pg autoscaler; lifecycle policies are not set. 4. We have 90+ hosts, most of which are Dell R720xd, most of the hard disks are 3.5 inches/5400 rpm/10TB Western Digital/, and most of the controllers are PERC H330 Mini; this is the current cluster status: cluster: id: f990db28-9604-4d49-9733-b17155887e3b health: HEALTH_OK services:   mon: 5 daemons, quorum cz-ceph-01,cz-ceph-02,cz-ceph-03,cz-ceph-07,cz-ceph-13   mgr: cz-ceph-01(active), standbys: cz-ceph-03, cz-ceph-02   osd: 1172 osds: 1172 up, 1172 in    rgw: 9 daemons active  data:    pools: 16 pools, 25752 pgs    objects: 2.23 G objects, 6.1 PiB    usage: 9.4 PiB used, 2.5 PiB / 12 PiB avail    pgs:      25686 active+clean                 64 active+clean+scrubbing+deep+repair                 2 active+clean+scrubbing+deep Best regards. Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/6 15:34 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: [ceph-users] Please guide us in identifying the cause ofthedata miss in EC pool Hi, Did the deep-scrub report any errors? (any inconsistencies should show errors after deep-scrubbing the PG.)   We're these errors fixed by the PG repair? Is is possible that you looked for the wrong PG or OSD when trying to list these objects with ceph-objectstore-tool? Were the PG autoscaler running at that time? Are you using S3 lifecycle policies that could have move this object to another placement pool and so another PG? Can you give details about this cluster? Hardware, disks, controller, etc. Cheers, Frédéric. De : wu_chulin(a)qq.com Envoyé : lundi 5 août 2024 10:09 À : Frédéric Nass Objet : Re:Re: [ceph-users] Please guide us in identifying the cause of thedata miss in EC pool Hi， Thank you for your reply. I apologize for the omission in the previous email. Please disregard the previous email and refer to this one instead.  After the failure, we executed the repair and deep-scrub command on some of the PGs that lost data, and the status was active+clean after completion, but the object still could not be retrieved. Our erasure code parameters are k=6, m=3. Theoretically, the data of three OSDs lost due to power failure should be recoverable. However, we stopped nine OSDs and exported the object list, but could not find the lost object information. What puzzled us was that some OSDs were not powered off and were still running, but their object lists did not have the information too. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/3 15:11 To:"wu_chulin"< wu_chulin(a)qq.com >;"ceph-users"< ceph-users(a)ceph.io >; Subject:Re: [ceph-users] Please guide us in identifying the cause of thedata miss in EC pool Hi, First thing that comes to mind when it comes to data unavailability or inconsistencies after a power outage is that some dirty data may have been lost along the IO path before reaching persistent storage. This can happen with non enterprise grade SSDs using non-persistent cache or with HDDs disk buffer if left enabled for example. With that said, have you tried to deep-scrub the PG from which you can't retrieve data? What's the status of this PG now? Did it recover? Regards, Frédéric. De : wu_chulin(a)qq.com Envoyé : mercredi 31 juillet 2024 05:49 À : ceph-users Objet : [ceph-users] Please guide us in identifying the cause of the data miss in EC pool Dear Ceph team:&nbsp; &nbsp; &nbsp;On July 13th at 4:55 AM, our Ceph cluster experienced a significant power outage in the data center, causing a large number of OSDs to power off and restart (total: 1172, down: 821). Approximately two hours later, all OSDs successfully started, and the cluster resumed its services. However, around 6 PM, the business department reported that some files, which had been successfully written (via the RGW service), were failing to download, and the number of such files was quite significant. Consequently, we began a series of investigations: 1. The incident occurred at 04:55. At 05:01, we executed noout, nobackfill, and norecover. At 06:22, we executed `ceph osd pause`. By 07:23, all OSDs were UP&amp;IN, and subsequently, we executed `ceph osd unpause`. 2. We randomly selected a problematic file and attempted to download it via the S3 API. The RGW returned "No such key". 3. The RGW logs showed op status=-2, http status=200. We also checked the upload logs, which indicated 2024-07-13 04:19:20.052, op status=0, http_status=200. 4. We set debug_rgw=20 and attempted to download the file again. It was found that a 4M chunk(this file is 64M) failed to get. 5. Using rados get for this chunk returned: "No such file or directory". 6. Setting debug_osd=20, we observed get_object_context: obc NOT found in cache. 7. Setting debug_bluestore=20, we saw get_onode oid xxx, key xxx != '0xfffffffffffffffeffffffffffffffff'o'. 8. We stopped the primary OSD and tried to get the file again, but the result was the same. The object’s corresponding PG state was active+recovery_wait+degraded. 9. Using ceph-objectstore-tool --op list &amp;&amp; --op log, we could not find the object information. The ceph-kvstore-tool rocksdb command also did not reveal anything new. 10. If an OSD had lost data, the PG state should have been unfound or inconsistency. 11. We started reanalyzing the startup logs of the OSDs related to the PG. The pool was set to erasure-code 6-3 with 9 OSDs. Six of these OSDs had restarted, and after peering, the PG state became ACTIVE. 12. We divided the lost files, and the upload time was before the failure occurred. The earliest upload time was around 1 am, and the successful upload records could be found in the RGW log 13. We have submitted an issue on the Ceph issue tracker:&nbsp;https://tracker.ceph.com/issues/66942, it includes the original logs needed for troubleshooting. However, four days have passed without any response. In desperation, we are sending this email, hoping that someone from the Ceph team can guide us as soon as possible. We are currently in a difficult situation and hope you can provide guidance. Thank you. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Please guide us inidentifying thecause ofthedata miss in EC pool
by Best Regards 08 Aug '24

08 Aug '24

Hi， Frédéric Nass Thank you for your continued attention and guidance. Let's analyze and verify this issue from different perspectives. The reason why we did not stop the investigation is that we tried to find other ways to avoid the losses caused by this sudden failure. Turning off the disk cache is the last option, of course, this operation will only be carried out after finding definite evidence. I also have a question that among the 9 OSDs, some have not been restarted. In theory, these OSDs should retain the object info(metadata,pg_log,etc.), even if the object cannot be recovered. I sorted out the OSD booting log where the object should be located and the PG peering process: OSD 494/1169/1057 has been in the running state, and osd.494 was the primary of the acting_set during the failure. However, no record of the object was found using `ceph-object-tool --op list or --op log` in, so the loss of data due to disk cache loss does not seem to be a complete explanation (perhaps there is some processing logic that we have not paid attention to). Best Regards, Woo wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 4:01 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us inidentifying thecause ofthedata miss in EC pool Hey Chulin, Looks clearer now. Non-persistent cache for KV metadata and Bluestore metadata certainly explains how data was lost without the cluster even noticing. What's unexpected is data staying for so long in the disks buffers and not being written to persistent sectors at all. Anyways, thank you for sharing your use case and investigation. It was nice chatting with you. If you can, share this in the ceph-user list. It will for sure benefit everyone in the community. Best regards, Frédéric. PS : Note that using min_size >= k + 1 on EC pools is recommended (so as min_size >= 2 on rep X3 pools) because you don't want to write data without any parity chunks. De : wu_chulin(a)qq.com Envoyé : mercredi 7 août 2024 11:30 À : Frédéric Nass Objet : Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us in identifying thecause ofthedata miss in EC pool Hi, Yes, after the file -> object -> PG -> OSD correspondence is found, the object record can be found on the specified OSD using the command `ceph-objectore-tool --op list ` The pool min_size is 6 The business department reported more than 30, but we proactively screened out more than 100. The upload time of the lost files was mainly distributed about 3 hours before the failure, and these files were successfully downloaded after being uploaded (RGW log). One OSD corresponds to one disk, and no separate space is allocated for WAL/DB. The HDD cache is the default (SATA is enabled by default), and the hard disk cache has not been forcibly turned off due to performance issues. The loss of OSD data due to the loss of hard disk cache was our initial inference, and the initial explanation provided to the business department was the same. When the cluster was restored, ceph reported 12 unfound objects, which is acceptable. After all, most devices were powered off abnormally, and it is difficult to ensure the integrity of all data. Up to now, our team have not located how the data was lost. In the past, when the hard disk hardware was damaged, either the OSD could not start because of damaged key data, or some objects were read incorrectly after the OSD started, which could be repaired. Now deep-scrub cannot find the problem, which may be related to the loss (or deletion) of object metadata. After all, deep-scrub needs the object list of the current PG. If those 9 OSDs do not have the object metadata information, deep-scrub does not know the existence of this object. wu_chulin(a)qq.com wu_chulin(a)qq.com Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/6 20:40 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: Re:Re: [ceph-users] Please guide us in identifying thecause ofthedata miss in EC pool That's interesting. Have you tried to correlate any existing retrievable object to PG id and OSD mapping in order to verify the presence of each of these object's shards using ceph-objectore-tool on each one of its acting OSDs, for a previously and successfully written S3 object. This would help verify that the command you've run trying to find the missing shard was good. Also what min_size is this pool using?    How many S3 objects like these were reported missing by the business unit? Have you or they made an inventory of unretrievable / missing objects? Are WAL / RocksDB collocated on HDDs only OSDs or are these OSDs using SSDs for WAL / RocksDB? Did you disable HDD buffers (also known as disk cache) as... HDD buffers are non-persistent. I know nothing about the intensity of your workloads but if your looking for a few tens or a few hundreds of unwritten s3 objects, there might be some situation with non-persistent cache (like volatile disk buffers) were ceph would consider the data to be written when it was actually not at the moment of the power outage, due to the use of non-persistent disk buffers. Especially if you kept writing data with less shards (min_size) than k+1 (no parity at all). That sound like a possibility. Also what I'm thinking right is... If you can identify which shard over 9 is wrong, then you may use ceph-objectore-tool or ceph-kvstore-tool to destroy this particular shard, then deep-scrub the PG so to detect the inexistent shard and have it rebuilt. Never tried this myself though. Best regards, Frédéric. De : wu_chulin(a)qq.com Envoyé : mardi 6 août 2024 12:15 À : Frédéric Nass Objet : Re:Re: Re:Re: [ceph-users] Please guide us in identifying the cause ofthedata miss in EC pool Hi,  Thank you for your attention to this matter. 1. Manually executing deep-scrub PG did not report any errors. I checked the OSD logs and did not see any errors detected or fixed by the OSD. Ceph health was also normal, and the OS where the OSD was located did not report any IO type errors. 2. At first, I also suspected whether the objects were distributed on these OSDs before the failure. I used ceph osd getmap <epoch_before_failure> and got "Error ENOENT: there is no map for epoch xxxxx". I am not sure if there are other ways to get the osdmap before the failure. After the failure, all OSDs joined the cluster, and the number was large (1000+). It was a bit difficult to find the OSD corresponding to the object without osdmap, so we didn't go into this direction. When the business department reported that the files could not be downloaded, I first opened the RGW debug log, found the mapping between file_name (64M) and oid (4M), then found the corresponding OSD according to ceph osd map <pool> <oid>, and then opened the OSD debug log to find the error information. 3. Our Ceph version is 13.2.10, which does not have pg autoscaler; lifecycle policies are not set. 4. We have 90+ hosts, most of which are Dell R720xd, most of the hard disks are 3.5 inches/5400 rpm/10TB Western Digital/, and most of the controllers are PERC H330 Mini; this is the current cluster status: cluster: id: f990db28-9604-4d49-9733-b17155887e3b health: HEALTH_OK services:   mon: 5 daemons, quorum cz-ceph-01,cz-ceph-02,cz-ceph-03,cz-ceph-07,cz-ceph-13   mgr: cz-ceph-01(active), standbys: cz-ceph-03, cz-ceph-02   osd: 1172 osds: 1172 up, 1172 in    rgw: 9 daemons active  data:    pools: 16 pools, 25752 pgs    objects: 2.23 G objects, 6.1 PiB    usage: 9.4 PiB used, 2.5 PiB / 12 PiB avail    pgs:      25686 active+clean                 64 active+clean+scrubbing+deep+repair                 2 active+clean+scrubbing+deep Best regards. Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/6 15:34 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: [ceph-users] Please guide us in identifying the cause ofthedata miss in EC pool Hi, Did the deep-scrub report any errors? (any inconsistencies should show errors after deep-scrubbing the PG.)   We're these errors fixed by the PG repair? Is is possible that you looked for the wrong PG or OSD when trying to list these objects with ceph-objectstore-tool? Were the PG autoscaler running at that time? Are you using S3 lifecycle policies that could have move this object to another placement pool and so another PG? Can you give details about this cluster? Hardware, disks, controller, etc. Cheers, Frédéric. De : wu_chulin(a)qq.com Envoyé : lundi 5 août 2024 10:09 À : Frédéric Nass Objet : Re:Re: [ceph-users] Please guide us in identifying the cause of thedata miss in EC pool Hi， Thank you for your reply. I apologize for the omission in the previous email. Please disregard the previous email and refer to this one instead.  After the failure, we executed the repair and deep-scrub command on some of the PGs that lost data, and the status was active+clean after completion, but the object still could not be retrieved. Our erasure code parameters are k=6, m=3. Theoretically, the data of three OSDs lost due to power failure should be recoverable. However, we stopped nine OSDs and exported the object list, but could not find the lost object information. What puzzled us was that some OSDs were not powered off and were still running, but their object lists did not have the information too. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/3 15:11 To:"wu_chulin"< wu_chulin(a)qq.com >;"ceph-users"< ceph-users(a)ceph.io >; Subject:Re: [ceph-users] Please guide us in identifying the cause of thedata miss in EC pool Hi, First thing that comes to mind when it comes to data unavailability or inconsistencies after a power outage is that some dirty data may have been lost along the IO path before reaching persistent storage. This can happen with non enterprise grade SSDs using non-persistent cache or with HDDs disk buffer if left enabled for example. With that said, have you tried to deep-scrub the PG from which you can't retrieve data? What's the status of this PG now? Did it recover? Regards, Frédéric. De : wu_chulin(a)qq.com Envoyé : mercredi 31 juillet 2024 05:49 À : ceph-users Objet : [ceph-users] Please guide us in identifying the cause of the data miss in EC pool Dear Ceph team:&nbsp; &nbsp; &nbsp;On July 13th at 4:55 AM, our Ceph cluster experienced a significant power outage in the data center, causing a large number of OSDs to power off and restart (total: 1172, down: 821). Approximately two hours later, all OSDs successfully started, and the cluster resumed its services. However, around 6 PM, the business department reported that some files, which had been successfully written (via the RGW service), were failing to download, and the number of such files was quite significant. Consequently, we began a series of investigations: 1. The incident occurred at 04:55. At 05:01, we executed noout, nobackfill, and norecover. At 06:22, we executed `ceph osd pause`. By 07:23, all OSDs were UP&amp;IN, and subsequently, we executed `ceph osd unpause`. 2. We randomly selected a problematic file and attempted to download it via the S3 API. The RGW returned "No such key". 3. The RGW logs showed op status=-2, http status=200. We also checked the upload logs, which indicated 2024-07-13 04:19:20.052, op status=0, http_status=200. 4. We set debug_rgw=20 and attempted to download the file again. It was found that a 4M chunk(this file is 64M) failed to get. 5. Using rados get for this chunk returned: "No such file or directory". 6. Setting debug_osd=20, we observed get_object_context: obc NOT found in cache. 7. Setting debug_bluestore=20, we saw get_onode oid xxx, key xxx != '0xfffffffffffffffeffffffffffffffff'o'. 8. We stopped the primary OSD and tried to get the file again, but the result was the same. The object’s corresponding PG state was active+recovery_wait+degraded. 9. Using ceph-objectstore-tool --op list &amp;&amp; --op log, we could not find the object information. The ceph-kvstore-tool rocksdb command also did not reveal anything new. 10. If an OSD had lost data, the PG state should have been unfound or inconsistency. 11. We started reanalyzing the startup logs of the OSDs related to the PG. The pool was set to erasure-code 6-3 with 9 OSDs. Six of these OSDs had restarted, and after peering, the PG state became ACTIVE. 12. We divided the lost files, and the upload time was before the failure occurred. The earliest upload time was around 1 am, and the successful upload records could be found in the RGW log 13. We have submitted an issue on the Ceph issue tracker:&nbsp;https://tracker.ceph.com/issues/66942, it includes the original logs needed for troubleshooting. However, four days have passed without any response. In desperation, we are sending this email, hoping that someone from the Ceph team can guide us as soon as possible. We are currently in a difficult situation and hope you can provide guidance. Thank you. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

mds damaged with preallocated inodes that are inconsistent with inotable
by zxcs 08 Aug '24

08 Aug '24

HI, Experts, we are running a cephfs with V16.2.*, and has multi active mds. Currently, we are hitting a mds fs cephfs mds.* id damaged. and this mds always complain “client *** loaded with preallocated inodes that are inconsistent with inotable” and the mds always suicide during replay. Could anyone please help here ? We really need you shed some light! Thanks lot ! xz

2 1

Can you return orphaned objects to a bucket?
by motaharesdq＠gmail.com 08 Aug '24

08 Aug '24

Hello everyone, Is there any way to return orphaned objects to a bucket? A large bucket was accidentally corrupted and emptied, but its objects do exist in rados. bi list and bucket check returns empty list and bucket fix don't do anything as expected. How can we index associated rados objects back to the bucket?

3 3

Re: Cephadm: unable to copy ceph.conf.new
by Eugen Block 07 Aug '24

07 Aug '24

Hi, please don't drop the ML from your response. Is this the first upgrade you're attempting or did previous upgrades work with the current config? > I wonder if can generate a new ssh configuration for the root user, > and then use that to upgrade to the fixed version. > The permissions will then be owned by root, which means we can't use > the ceph user, no? I do remember having an issue with non-root user on a customer cluster, but IIRC it was because of insufficient sudo permissions. In the end, they switched to root user, and there haven't been any issues since, at least nobody reported anything to me. Do you mind sharing your sudo config for the ceph user? Thanks, Eugen Zitat von Magnus Larsen <magnusfynbo(a)hotmail.com>: > Hi, > > We do have client-keyring with the label: > # ceph orch client-keyring ls > ENTITY PLACEMENT MODE OWNER PATH > client.admin label:_admin rw------- 0:0 > /etc/ceph/ceph.client.admin.keyring > > And the SSH-config is also correct (verified just now) - though we > use ceph as the user, not the default root, > which works normally, except that we can't upgrade until we get the > fix in... which is in the next upgrade :< > > I wonder if can generate a new ssh configuration for the root user, > and then use that to upgrade to the fixed version. > The permissions will then be owned by root, which means we can't use > the ceph user, no? > > ref: https://docs.ceph.com/en/octopus/cephadm/operations/#ssh-configuration > > Thanks! > Magnus Larsen > > ________________________________ > Fra: Eugen Block <eblock(a)nde.ag> > Sendt: 7. august 2024 09:15 > Til: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Emne: [ceph-users] Re: Cephadm: unable to copy ceph.conf.new > > Hi, > > I commented a similar issue a couple of months ago: > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/IQX2VXA6QQ… > > Can you check if that applies to your cluster? > > Zitat von Magnus Larsen <magnusfynbo(a)hotmail.com>: > >> Hi Ceph-users! >> >> Ceph version: ceph version 17.2.6 >> (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) >> Using cephadm to orchestrate the Ceph cluster >> >> I’m running into https://tracker.ceph.com/issues/59189, which is >> fixed in next version—quincy 17.2.7—via >> https://github.com/ceph/ceph/pull/50906 >> >> But I am unable to upgrade to the fixed version because of that bug >> >> When I try to upgrade (using “ceph orch upgrade start –image >> internal_mirror/ceph:v17.2.7”), we see the same error message: >> executing _write_files((['dkcphhpcadmin01', 'dkcphhpcmgt028', >> 'dkcphhpcmgt029', 'dkcphhpcmgt031', 'dkcphhpcosd033', >> 'dkcphhpcosd034', 'dkcphhpcosd035', 'dkcphhpcosd036', >> 'dkcphhpcosd037', 'dkcphhpcosd038', 'dkcphhpcosd039', >> 'dkcphhpcosd040', 'dkcphhpcosd041', 'dkcphhpcosd042', >> 'dkcphhpcosd043', 'dkcphhpcosd044'],)) failed. Traceback (most >> recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line >> 240, in _write_remote_file conn = await >> self._remote_connection(host, addr) File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp >> await source.run(srcpath) File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run >> self.handle_error(exc) File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in >> handle_error raise exc from None File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run >> await self._send_files(path, b'') File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in >> _send_files self.handle_error(exc) File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in >> handle_error raise exc from None File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in >> _send_files await self._send_file(srcpath, dstpath, attrs) File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in >> _send_file await self._make_cd_request(b'C', attrs, size, srcpath) >> File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in >> _make_cd_request self._fs.basename(path)) File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in >> make_request raise exc asyncssh.sftp.SFTPFailure: scp: >> /tmp/etc/ceph/ceph.conf.new: Permission denied During handling of >> the above exception, another exception occurred: Traceback (most >> recent call last): File "/usr/share/ceph/mgr/cephadm/utils.py", line >> 79, in do_work return f(*arg) File >> "/usr/share/ceph/mgr/cephadm/serve.py", line 1088, in _write_files >> self._write_client_files(client_files, host) File >> "/usr/share/ceph/mgr/cephadm/serve.py", line 1107, in >> _write_client_files self.mgr.ssh.write_remote_file(host, path, >> content, mode, uid, gid) File "/usr/share/ceph/mgr/cephadm/ssh.py", >> line 261, in write_remote_file >> self.mgr.wait_async(self._write_remote_file( File >> "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async >> return self.event_loop.get_result(coro) File >> "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result return >> asyncio.run_coroutine_threadsafe(coro, self._loop).result() File >> "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result >> return self.__get_result() File >> "/lib64/python3.6/concurrent/futures/_base.py", line 384, in >> __get_result raise self._exception File >> "/usr/share/ceph/mgr/cephadm/ssh.py", line 249, in >> _write_remote_file logger.exception(msg) >> orchestrator._interface.OrchestratorError: Unable to write >> dkcphhpcmgt028:/etc/ceph/ceph.conf: scp: >> /tmp/etc/ceph/ceph.conf.new: Permission denied >> >> We were thinking about removing the keyring from the Ceph >> orchestrator >> (https://docs.ceph.com/en/latest/cephadm/operations/#putting-a-keyring-under…) >> which would then make Ceph not try to copy over a new ceph.conf, >> alleviating the problem >> (https://docs.ceph.com/en/latest/cephadm/operations/#client-keyrings-and-con…) >> but in doing so, Ceph will kindly remove the key from all nodes >> (https://docs.ceph.com/en/latest/cephadm/operations/#disabling-management-of…) >> leaving us without the admin keyring. So that doesn’t sound like a >> path we want to take :S >> >> Does anybody know how to get around this issue, so I can get to >> version where the issue fixed for good? >> >> Thanks, >> Magnus >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

What's the best way to add numerous OSDs?
by Fabien Sirjean 07 Aug '24

07 Aug '24

Hello everyone, We need to add 180 20TB OSDs to our Ceph cluster, which currently consists of 540 OSDs of identical size (replicated size 3). I'm not sure, though: is it a good idea to add all the OSDs at once? Or is it better to add them gradually? The idea is to minimize the impact of rebalancing on the performance of CephFS, which is used in production. Thanks in advance for your opinions and feedback 🙂 Wishing you a great summer, Fabien

4 3

RGW bucket notifications stop working after a while and blocking requests
by Florian Schwab 06 Aug '24

06 Aug '24

Hi, we just set up 2 new ceph clusters (using rook). To do some processing of the user activity we configured a topic that sends events to Kafka. After 5-12 hours this stops working with a 503 SlowDown response: debug 2024-08-02T09:17:58.205+0000 7ff4359ad700 1 req 13681579273117692719 0.005000019s ERROR: failed to reserve notification on queue: private.rgw. error: -28 First thought would be that the queue is full but up to this point see messages coming into Kafka and without much activity on the RGW itself (only a few requests against the S3 API) so it can’t be a load issue. What helps is to remove the notification configuration on the buckets (put-bucket-notification-configuration). If we directly re-add the previous notification configuration it also continuous working for a few hours before failing again with the same error/behaviour. We haven’t been able to reproduce this if we disable persistence for the topic so it looks like it is related to the persistence option - otherwise there would be also no queuing of the event for sending to Kafka. This also suggests that the issue is not with Kafka - this is also what we suspected first e.g. it can’t handle the amount of messages etc. Does anyone else have or had this issue and found the cause or a suggestion on how to best continue debugging? Are there detailed metrics etc. on the size and usage of the event queue? Here is the configuration for the topic and for a bucket: $ radosgw-admin topic list { "topics": [ { "user": "", "name": "private.rgw", "dest": { "push_endpoint": "kafka://rgw-sasl-kafka-user:XXX@kafka-kafka-bootstrap.kafka.svc:9094/private.rgw?sasl.mechanism=SCRAM-SHA-512&mechanism=SCRAM-SHA-512", "push_endpoint_args": "OpaqueData=&Version=2010-03-31&kafka-ack-level=broker&persistent=false&push-endpoint=kafka://rgw-sasl-kafka-user:XXX@kafka-kafka-bootstrap.kafka.svc:9094/private.rgw?sasl.mechanism=SCRAM-SHA-512&mechanism=SCRAM-SHA-512&use-ssl=true&verify-ssl=true", "push_endpoint_topic": "private.rgw", "stored_secret": true, "persistent": true }, "arn": "arn:aws:sns:ceph-objectstore::private.rgw", "opaqueData": "" } ] } $ aws s3api get-bucket-notification-configuration --bucket=XXX { "TopicConfigurations": [ { "Id": “my-id", "TopicArn": "arn:aws:sns:ceph-objectstore::private.rgw", "Events": [ "s3:ObjectCreated:*", "s3:ObjectRemoved:*" ] } ] } Thank you for any input to solve this! Cheers, Florian

2 3

Ceph Developer Summit (Tentacle) Aug 12-19
by Noah Lehman 06 Aug '24

06 Aug '24

Hi Ceph users, The next Ceph Developer Summit is happening virtually from August 12 – 19, 2024 adn we want to see you there. The focus of the summit will include planning around our next release, Tentacle, and everyone in our community is welcome to participate! Learn more and RSVP here: https://ceph.io/en/community/events/2024/ceph-developer-summit-tentacle/ Best, Noah

1 0

Recovering from total mon loss and backing up lockbox secrets
by Boris 06 Aug '24

06 Aug '24

Hi, I am in the process of creating disaster recovery documentation and I have two topics where I am not sure how to do it or even if it is possible. Is it possible to recover from a 100% mon data loss? Like all mons fail and the actual mon data is not recoverable. In my head I would thing that I can just create new mons with the same cluster ID and then start everything. The OSDs still have their PGs and data and after some period of time everything will be ok again. But then I thought that we use dmcrypt in ceph and I would need to somehow backup all the keys to some offsite location. So here are my questions: - How do I backup the lockbox secrets? - Do I need to backup the whole mon data, and if so how can I do it? Cheers Boris

2 1

Osds going down/flapping after Luminous to Nautilus upgrade part 1
by Mark Kirkwood 06 Aug '24

06 Aug '24

We have upgraded one of our Ceph clusters to Nautilus. We have run into 2 issues that are causing osds to flap. I'll cover the 1st one here, this one we solved but it raises an interesting question that might bear on the 2nd one (will post that next). After upgrading deb packages to Nautilus and restarting the mons and mgrs we worked through restarting the osds. We started to see some of them flap and saw this in the osd log (many times): 2024-07-31 11:03:33.264 7f22ab6e0700 0 --1- [v2:[2404:130:8020:5::73]:6820/220732,v1:[2404:130:8020:5::73]:6821/220732] >> v1:[2404:130:8020:5::103]:6909/2987374 conn(0x555a5bbbb180 0x555a4172c800 :-1 s=OPENED pgs=144 cs=3 l=0).fault initiating reconnect And later (usually a single line): 2404:130:8020:5::135]:6903/2933993 conn(0x555ab7b03200 0x555a099fa000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=154 cs=4 l=0).handle_connect_reply_2 connect got BADAUTHORIZER Examining the code showed: markir@zmori:/download/ceph/src/ceph$ find . -type f -exec grep -l "initiating reconnect" {} \; ./src/msg/simple/Pipe.cc ./src/msg/async/ProtocolV1.cc ./src/msg/async/ProtocolV2.cc markir@zmori:/download/ceph/src/ceph$ vi src/msg/async/ProtocolV2.cc markir@zmori:/download/ceph/src/ceph$ find . -type f -exec grep -l "got BADAUTHORIZER" {} \; ./src/msg/simple/Pipe.cc ./src/msg/async/ProtocolV1.cc Which led us to suspect that the osds were using the v1 msgr protocol (ceph osd dump seems to validate this). We hoped that once we enabled the v2 msgr that this error would vanish. This appears to have happened. So my question is this: looks like there is something wrong with communications via v1 protocol post upgrade - is that expected? Regards Mark

2 1