Hello folks. I got a failed cluster recovery job and the cluster is: 2PB EC 8+2 pool used as RGW S3 storage. (14.2.16 nautilus) I fixed most of the issues and rescued a lot of failed OSD's but some OSD's are not recoverable and the cluster size is huge, recovery takes time. Fixing all the issues is impossible because missing OSD's and running PG repair take weeks. I did my best and the current state is enough for starting data export. I only need to export the healthy data first and then I'm gonna destroy-rebuild the cluster. I tried to export data by using rclone and it runs good at the beginning but when the radosgw service hits an unfound object it slows down and it freezes. Because of this issue I can not export data by using rclone+radosgw I'm using these parameters for the export:
rclone copy --files-from=object-list.txt --progress --checkers=8 --transfers=8 --no-traverse \ --no-update-modtime --ignore-checksum --ignore-size --no-gzip-encoding --s3-no-check-bucket \ --buffer-size=1G --low-level-retries=1 --retries=1 --retries-sleep=100ms --timeout=30s \ --contimeout=30s --ignore-errors --ignore-existing --log-file=export-rclone.log --log-level=INFO
Current state is:
# ceph health detail (14.2.16 nautilus) HEALTH_ERR 3303 large omap objects; 1460729/1238267531 objects unfound (0.118%); Possible data damage: 2 pgs recovery_unfound, 1 pg backfill_unfound; Degraded data redundancy: 5487055/12315635595 objects degraded (0.045%), 7 pgs degraded, 7 pgs undersized; 1987 pgs not deep-scrubbed in time; 1824 pgs not scrubbed in time LARGE_OMAP_OBJECTS 3303 large omap objects 165 large objects found in pool 'xyz.rgw.log' 3138 large objects found in pool 'xyz.rgw.buckets.index' Search the cluster log for 'Large omap object found' for more details. OBJECT_UNFOUND 1460729/1238267531 objects unfound (0.118%) pg 56.73a has 99125 unfound objects pg 56.470 has 572028 unfound objects pg 56.3a9 has 564371 unfound objects pg 56.346 has 7871 unfound objects pg 56.36 has 1 unfound objects pg 56.1e2 has 108269 unfound objects pg 56.204 has 109064 unfound objects PG_DAMAGED Possible data damage: 2 pgs recovery_unfound, 1 pg backfill_unfound pg 56.36 is active+backfill_unfound+undersized+degraded+remapped, acting [166,36,7,46,152,193,128,223,80,2147483647], 1 unfound pg 56.3a9 is active+recovery_unfound+undersized+degraded+remapped, acting [153,69,177,156,49,167,2,2147483647,2147483647,31], 564371 unfound pg 56.470 is active+recovery_unfound+undersized+degraded+remapped, acting [135,2147483647,2147483647,176,37,85,65,48,183,224], 572028 unfound PG_DEGRADED Degraded data redundancy: 5487055/12315635595 objects degraded (0.045%), 7 pgs degraded, 7 pgs undersized pg 56.36 is stuck undersized for 47661.320358, current state active+backfill_unfound+undersized+degraded+remapped, last acting [166,36,7,46,152,193,128,223,80,2147483647] pg 56.1e2 is stuck undersized for 47660.312165, current state active+recovering+undersized+degraded+remapped, last acting [2147483647,127,77,140,167,195,220,131,113,31] pg 56.204 is stuck undersized for 47660.325926, current state active+recovering+undersized+degraded+remapped, last acting [2147483647,42,173,17,64,189,22,113,136,158] pg 56.346 is stuck undersized for 47660.335077, current state active+recovering+undersized+degraded+remapped, last acting [63,108,53,140,132,2147483647,183,171,0,98] pg 56.3a9 is stuck undersized for 4403604.202577, current state active+recovery_unfound+undersized+degraded+remapped, last acting [153,69,177,156,49,167,2,2147483647,2147483647,31] pg 56.470 is stuck undersized for 47661.762140, current state active+recovery_unfound+undersized+degraded+remapped, last acting [135,2147483647,2147483647,176,37,85,65,48,183,224] pg 56.73a is stuck undersized for 47656.126741, current state active+recovering+undersized+degraded+remapped, last acting [2147483647,189,85,155,170,106,40,25,222,121] PG_NOT_DEEP_SCRUBBED 1987 pgs not deep-scrubbed in time
I considered "bucket check" but each bucket has 10M objects and 100TB of data. It will take days-weeks and it may fail because of the cluster state.
"rgw.main": { "size": 106443540756257, "size_actual": 106463698989056, "size_utilized": 106443540756257, "size_kb": 103948770270, "size_kb_actual": 103968456044, "size_kb_utilized": 103948770270, "num_objects": 9855872 },
I'm thinking of writing a rados exporter script and I need to do this: 1. Read bucket index and have a list of objects. 2. Check the bucket data by each object and eliminate the unfound objects and generate a clean list with healthy objects 3. Export the data directly from rados to an NFS share by combining multipart data. My questions are: 1. Are there any other ways to use radosgw+rclone for fast export from ERROR state cluster? 2. If not, then do you know any rados export tool I can use? I really need your advice. What is the best approach to this? Thank you.