March 2023 - ceph-users - lists.ceph.io

Unbalanced OSDs when pg_autoscale enabled

by 郑亮

Hi all, I have a 9 node cluster running *Pacific 16.2.10*. OSDs live on 9 of the nodes with each one having 4 x 1.8T ssd and 8 x 10.9T hdd for a total of 108 OSDs. We create three crush roots as belows. 1. The hdds (8x9=72) of all nodes form a large crush root, which is used as a data pool, and object storage and cephfs share this crush root. 2. Take 3 ssds from the 4 ssds of each node as rbd block storage. 3. An ssd on each remaining node is used as an index pool for cephfs and object storage. [root@node01 smd]# ceph osd treeID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -92 15.71910 root root-1c31624a-ad18-445e-8e42-86b71c1fd76f -112 1.74657 host node01-fa2cdf3e-7212-4b5f-b62a-3ab1e803547f 13 ssd 1.74657 osd.13 up 1.00000 1.00000 -103 1.74657 host node02-4e232f27-fe4b-4d0e-bd2a-67d5006a0cdd 34 ssd 1.74657 osd.34 up 1.00000 1.00000 -109 1.74657 host node03-3ae63d7a-9f65-4bea-b2ba-ff3fe342753d 28 ssd 1.74657 osd.28 up 1.00000 1.00000 -118 1.74657 host node04-37a3f92a-f6d8-41f9-a774-3069fc2f50b8 54 ssd 1.74657 osd.54 up 1.00000 1.00000 -106 1.74657 host node05-f667fa27-cc13-4b93-ad56-5dc4c31ffd77 53 ssd 1.74657 osd.53 up 1.00000 1.00000 -91 1.74657 host node06-3808c8f6-8e10-47c7-8456-62c1e0e800ed 61 ssd 1.74657 osd.61 up 1.00000 1.00000 -97 1.74657 host node07-78216b0d-0999-44e8-905d-8737a5f6f51f 50 ssd 1.74657 osd.50 up 1.00000 1.00000 -115 1.74657 host node08-947bd556-fb06-497d-8f2c-c4a679d2b06f 86 ssd 1.74657 osd.86 up 1.00000 1.00000 -100 1.74657 host node09-d9ae9046-0716-454f-ba0c-b03cf9986ba8 85 ssd 1.74657 osd.85 up 1.00000 1.00000 -38 785.80701 root root-6041a4dc-7c9a-44ed-999c-a847cca81012 -85 87.31189 host node01-e5646053-2cf8-4ba5-90d5-bb1a63b1234c 1 hdd 10.91399 osd.1 up 1.00000 1.00000 22 hdd 10.91399 osd.22 up 0.90002 1.00000 31 hdd 10.91399 osd.31 up 1.00000 1.00000 51 hdd 10.91399 osd.51 up 1.00000 1.00000 60 hdd 10.91399 osd.60 up 1.00000 1.00000 70 hdd 10.91399 osd.70 up 1.00000 1.00000 78 hdd 10.91399 osd.78 up 1.00000 1.00000 96 hdd 10.91399 osd.96 up 1.00000 1.00000 -37 87.31189 host node02-be9925fd-60de-4147-81eb-720d7145715f 9 hdd 10.91399 osd.9 up 1.00000 1.00000 19 hdd 10.91399 osd.19 up 1.00000 1.00000 29 hdd 10.91399 osd.29 up 1.00000 1.00000 47 hdd 10.91399 osd.47 up 1.00000 1.00000 56 hdd 10.91399 osd.56 up 1.00000 1.00000 65 hdd 10.91399 osd.65 up 1.00000 1.00000 88 hdd 10.91399 osd.88 up 1.00000 1.00000 98 hdd 10.91399 osd.98 up 1.00000 1.00000 -52 87.31189 host node03-7828653d-6033-4e88-92b0-d8709b0ab218 2 hdd 10.91399 osd.2 up 1.00000 1.00000 30 hdd 10.91399 osd.30 up 1.00000 1.00000 40 hdd 10.91399 osd.40 up 1.00000 1.00000 48 hdd 10.91399 osd.48 up 1.00000 1.00000 58 hdd 10.91399 osd.58 up 1.00000 1.00000 74 hdd 10.91399 osd.74 up 1.00000 1.00000 83 hdd 10.91399 osd.83 up 1.00000 1.00000 92 hdd 10.91399 osd.92 up 1.00000 1.00000 -46 87.31189 host node04-e986c3fc-a21b-44ff-9b02-b60b82ee63d7 12 hdd 10.91399 osd.12 up 1.00000 1.00000 23 hdd 10.91399 osd.23 up 1.00000 1.00000 32 hdd 10.91399 osd.32 up 1.00000 1.00000 43 hdd 10.91399 osd.43 up 1.00000 1.00000 52 hdd 10.91399 osd.52 up 1.00000 1.00000 71 hdd 10.91399 osd.71 up 1.00000 1.00000 95 hdd 10.91399 osd.95 up 1.00000 1.00000 104 hdd 10.91399 osd.104 up 1.00000 1.00000 -88 87.31189 host node05-fe31d85f-b3b9-4393-b24b-030dbcdfacea 3 hdd 10.91399 osd.3 up 1.00000 1.00000 24 hdd 10.91399 osd.24 up 1.00000 1.00000 33 hdd 10.91399 osd.33 up 1.00000 1.00000 45 hdd 10.91399 osd.45 up 1.00000 1.00000 69 hdd 10.91399 osd.69 up 1.00000 1.00000 79 hdd 10.91399 osd.79 up 1.00000 1.00000 89 hdd 10.91399 osd.89 up 1.00000 1.00000 99 hdd 10.91399 osd.99 up 1.00000 1.00000 -55 87.31189 host node06-6f16ba4b-0082-472a-b243-b1a058070918 5 hdd 10.91399 osd.5 up 1.00000 1.00000 15 hdd 10.91399 osd.15 up 1.00000 1.00000 25 hdd 10.91399 osd.25 up 1.00000 1.00000 44 hdd 10.91399 osd.44 up 1.00000 1.00000 63 hdd 10.91399 osd.63 up 1.00000 1.00000 72 hdd 10.91399 osd.72 up 1.00000 1.00000 81 hdd 10.91399 osd.81 up 1.00000 1.00000 91 hdd 10.91399 osd.91 up 1.00000 1.00000 -43 87.31189 host node07-5dee846a-2814-4e04-bcfd-ff689d49795c 7 hdd 10.91399 osd.7 up 1.00000 1.00000 37 hdd 10.91399 osd.37 up 1.00000 1.00000 46 hdd 10.91399 osd.46 up 1.00000 1.00000 55 hdd 10.91399 osd.55 up 1.00000 1.00000 64 hdd 10.91399 osd.64 up 1.00000 1.00000 84 hdd 10.91399 osd.84 up 1.00000 1.00000 94 hdd 10.91399 osd.94 up 1.00000 1.00000 102 hdd 10.91399 osd.102 up 1.00000 1.00000 -58 87.31189 host node08-2d6b7ab3-2067-4e94-b77c-24d6e626e396 6 hdd 10.91399 osd.6 up 1.00000 1.00000 18 hdd 10.91399 osd.18 up 1.00000 1.00000 41 hdd 10.91399 osd.41 up 1.00000 1.00000 49 hdd 10.91399 osd.49 up 1.00000 1.00000 59 hdd 10.91399 osd.59 up 1.00000 1.00000 68 hdd 10.91399 osd.68 up 1.00000 1.00000 97 hdd 10.91399 osd.97 up 1.00000 1.00000 105 hdd 10.91399 osd.105 up 1.00000 1.00000 -49 87.31189 host node09-e6bf0642-e3b4-48f0-9d0e-3d87ceacced8 8 hdd 10.91399 osd.8 up 1.00000 1.00000 17 hdd 10.91399 osd.17 up 1.00000 1.00000 27 hdd 10.91399 osd.27 up 1.00000 1.00000 38 hdd 10.91399 osd.38 up 1.00000 1.00000 57 hdd 10.91399 osd.57 up 1.00000 1.00000 67 hdd 10.91399 osd.67 up 1.00000 1.00000 93 hdd 10.91399 osd.93 up 1.00000 1.00000 103 hdd 10.91399 osd.103 up 1.00000 1.00000 -8 47.16115 root root-ea7a1878-722e-49d5-8a91-c618a6aefe29 -13 5.24013 host node01-4c465825-6bd1-42a5-b087-51a82cb2865c 0 ssd 1.74657 osd.0 up 0.95001 1.00000 11 ssd 1.74699 osd.11 up 1.00000 1.00000 16 ssd 1.74657 osd.16 up 1.00000 1.00000 -22 5.24013 host node02-0e3e418d-9129-44ee-8453-53680171270e 4 ssd 1.74657 osd.4 up 1.00000 1.00000 14 ssd 1.74657 osd.14 up 0.95001 1.00000 82 ssd 1.74699 osd.82 up 1.00000 1.00000 -28 5.24013 host node03-274397b4-8559-4a17-a9eb-1c16920ad432 10 ssd 1.74657 osd.10 up 1.00000 1.00000 76 ssd 1.74657 osd.76 up 1.00000 1.00000 100 ssd 1.74699 osd.100 up 0.95001 1.00000 -31 5.24013 host node04-df05c3cc-08ef-426e-85b8-088cb8c1b4e2 20 ssd 1.74657 osd.20 up 1.00000 1.00000 39 ssd 1.74657 osd.39 up 0.95001 1.00000 90 ssd 1.74699 osd.90 up 0.95001 1.00000 -25 5.24013 host node05-d07e7e92-8290-49ba-b09c-ad34bddb1eae 26 ssd 1.74657 osd.26 up 0.95001 1.00000 62 ssd 1.74699 osd.62 up 1.00000 1.00000 73 ssd 1.74657 osd.73 up 1.00000 1.00000 -16 5.24013 host node06-a7a40371-a128-4ab1-90a2-62c99e040036 42 ssd 1.74657 osd.42 up 0.95001 1.00000 75 ssd 1.74657 osd.75 up 0.95001 1.00000 107 ssd 1.74699 osd.107 up 1.00000 1.00000 -7 5.24013 host node07-971dee6e-dec4-4c0a-86d1-54d0b23832bd 21 ssd 1.74699 osd.21 up 1.00000 1.00000 66 ssd 1.74657 osd.66 up 0.95001 1.00000 77 ssd 1.74657 osd.77 up 1.00000 1.00000 -19 5.24013 host node08-9a47282b-9530-4cc7-9e29-7f6c0b2f5184 35 ssd 1.74699 osd.35 up 1.00000 1.00000 80 ssd 1.74657 osd.80 up 1.00000 1.00000 101 ssd 1.74657 osd.101 up 1.00000 1.00000 -34 5.24013 host node09-f09a8013-2426-4ced-b9c7-02c06ca9d6fc 36 ssd 1.74699 osd.36 up 0.95001 1.00000 87 ssd 1.74657 osd.87 up 1.00000 1.00000 106 ssd 1.74657 osd.106 up 1.00000 1.00000 After ceph has been running normally for a period of time, I found that the distribution of osd data is seriously uneven, and autoscale PGs were turned on in my envs. My hdds are identical, but utilization ranges from 6.14% to 74.35% root@node09:/# ceph osd df | grep hdd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1 hdd 10.91399 1.00000 11 TiB 5.3 TiB 5.2 TiB 15 KiB 53 GiB 5.7 TiB 48.15 1.38 174 up 22 hdd 10.91399 0.90002 11 TiB 8.1 TiB 8.0 TiB 12 KiB 71 GiB 2.8 TiB 74.35 2.13 160 up 31 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 13 KiB 27 GiB 8.0 TiB 27.14 0.78 179 up 51 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 9 KiB 36 GiB 7.2 TiB 34.12 0.98 173 up 60 hdd 10.91399 1.00000 11 TiB 973 GiB 962 GiB 14 KiB 11 GiB 10 TiB 8.71 0.25 167 up 70 hdd 10.91399 1.00000 11 TiB 2.0 TiB 2.0 TiB 11 KiB 3.9 GiB 8.9 TiB 18.42 0.53 173 up 78 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 12 KiB 28 GiB 7.9 TiB 27.18 0.78 173 up 96 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 10 KiB 36 GiB 7.2 TiB 34.15 0.98 176 up 9 hdd 10.91399 1.00000 11 TiB 2.8 TiB 2.8 TiB 16 KiB 13 GiB 8.1 TiB 25.38 0.73 178 up 19 hdd 10.91399 1.00000 11 TiB 3.8 TiB 3.8 TiB 13 KiB 43 GiB 7.1 TiB 35.04 1.00 177 up 29 hdd 10.91399 1.00000 11 TiB 5.1 TiB 5.0 TiB 14 KiB 38 GiB 5.9 TiB 46.38 1.33 182 up 47 hdd 10.91399 1.00000 11 TiB 686 GiB 684 GiB 9 KiB 1.4 GiB 10 TiB 6.14 0.18 172 up 56 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 12 KiB 36 GiB 7.2 TiB 34.16 0.98 174 up 65 hdd 10.91399 1.00000 11 TiB 5.9 TiB 5.9 TiB 7 KiB 54 GiB 5.0 TiB 54.27 1.55 183 up 88 hdd 10.91399 1.00000 11 TiB 4.5 TiB 4.4 TiB 12 KiB 44 GiB 6.4 TiB 41.17 1.18 173 up 98 hdd 10.91399 1.00000 11 TiB 7.3 TiB 7.2 TiB 13 KiB 56 GiB 3.7 TiB 66.49 1.90 179 up 2 hdd 10.91399 1.00000 11 TiB 1.5 TiB 1.5 TiB 12 KiB 18 GiB 9.4 TiB 14.02 0.40 177 up 30 hdd 10.91399 1.00000 11 TiB 4.4 TiB 4.4 TiB 10 KiB 37 GiB 6.5 TiB 40.29 1.15 180 up 40 hdd 10.91399 1.00000 11 TiB 1.4 TiB 1.4 TiB 10 KiB 11 GiB 9.5 TiB 13.14 0.38 172 up 48 hdd 10.91399 1.00000 11 TiB 1.4 TiB 1.4 TiB 12 KiB 11 GiB 9.5 TiB 13.17 0.38 168 up 58 hdd 10.91399 1.00000 11 TiB 7.5 TiB 7.4 TiB 14 KiB 70 GiB 3.5 TiB 68.31 1.96 182 up 74 hdd 10.91399 1.00000 11 TiB 6.0 TiB 6.0 TiB 10 KiB 61 GiB 4.9 TiB 55.15 1.58 181 up 83 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 22 KiB 19 GiB 8.7 TiB 20.13 0.58 174 up 92 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 11 KiB 27 GiB 8.0 TiB 27.15 0.78 171 up 12 hdd 10.91399 1.00000 11 TiB 5.1 TiB 5.0 TiB 9 KiB 38 GiB 5.9 TiB 46.37 1.33 176 up 23 hdd 10.91399 1.00000 11 TiB 4.4 TiB 4.4 TiB 22 KiB 37 GiB 6.5 TiB 40.24 1.15 173 up 32 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 10 KiB 19 GiB 8.7 TiB 20.13 0.58 176 up 43 hdd 10.91399 1.00000 11 TiB 2.0 TiB 2.0 TiB 10 KiB 4.4 GiB 8.9 TiB 18.41 0.53 176 up 52 hdd 10.91399 1.00000 11 TiB 3.6 TiB 3.6 TiB 12 KiB 29 GiB 7.3 TiB 33.27 0.95 171 up 71 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 15 KiB 27 GiB 8.0 TiB 27.13 0.78 170 up 95 hdd 10.91399 1.00000 11 TiB 5.4 TiB 5.3 TiB 10 KiB 60 GiB 5.6 TiB 49.04 1.40 173 up104 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 8 KiB 36 GiB 7.2 TiB 34.13 0.98 171 up 3 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 9 KiB 19 GiB 8.7 TiB 20.14 0.58 173 up 24 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 9 KiB 19 GiB 8.7 TiB 20.12 0.58 170 up 33 hdd 10.91399 1.00000 11 TiB 6.8 TiB 6.7 TiB 14 KiB 69 GiB 4.1 TiB 62.15 1.78 181 up 45 hdd 10.91399 1.00000 11 TiB 2.9 TiB 2.8 TiB 10 KiB 20 GiB 8.0 TiB 26.27 0.75 178 up 69 hdd 10.91399 1.00000 11 TiB 2.9 TiB 2.8 TiB 15 KiB 20 GiB 8.0 TiB 26.25 0.75 173 up 79 hdd 10.91399 1.00000 11 TiB 5.9 TiB 5.9 TiB 28 KiB 54 GiB 5.0 TiB 54.25 1.55 178 up 89 hdd 10.91399 1.00000 11 TiB 2.1 TiB 2.1 TiB 16 KiB 11 GiB 8.8 TiB 19.25 0.55 173 up 99 hdd 10.91399 1.00000 11 TiB 4.5 TiB 4.4 TiB 13 KiB 44 GiB 6.4 TiB 41.14 1.18 174 up 5 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 14 KiB 19 GiB 8.7 TiB 20.12 0.58 172 up 15 hdd 10.91399 1.00000 11 TiB 7.3 TiB 7.2 TiB 7 KiB 56 GiB 3.7 TiB 66.48 1.90 177 up 25 hdd 10.91399 1.00000 11 TiB 1.4 TiB 1.4 TiB 11 KiB 10 GiB 9.5 TiB 13.13 0.38 176 up 44 hdd 10.91399 1.00000 11 TiB 3.1 TiB 3.0 TiB 9 KiB 34 GiB 7.9 TiB 28.03 0.80 179 up 63 hdd 10.91399 1.00000 11 TiB 2.9 TiB 2.8 TiB 10 KiB 20 GiB 8.0 TiB 26.26 0.75 178 up 72 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 12 KiB 36 GiB 7.2 TiB 34.13 0.98 180 up 81 hdd 10.91399 1.00000 11 TiB 6.7 TiB 6.6 TiB 11 KiB 62 GiB 4.2 TiB 61.30 1.76 181 up 91 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 16 KiB 19 GiB 8.7 TiB 20.11 0.58 176 up 7 hdd 10.91399 1.00000 11 TiB 4.6 TiB 4.5 TiB 13 KiB 52 GiB 6.3 TiB 42.03 1.20 180 up 37 hdd 10.91399 1.00000 11 TiB 5.3 TiB 5.2 TiB 14 KiB 52 GiB 5.7 TiB 48.16 1.38 182 up 46 hdd 10.91399 1.00000 11 TiB 784 GiB 775 GiB 14 KiB 8.8 GiB 10 TiB 7.01 0.20 173 up 55 hdd 10.91399 1.00000 11 TiB 5.9 TiB 5.9 TiB 20 KiB 54 GiB 5.0 TiB 54.25 1.55 177 up 64 hdd 10.91399 1.00000 11 TiB 3.6 TiB 3.6 TiB 18 KiB 28 GiB 7.3 TiB 33.23 0.95 176 up 84 hdd 10.91399 1.00000 11 TiB 784 GiB 775 GiB 10 KiB 9.0 GiB 10 TiB 7.01 0.20 176 up 94 hdd 10.91399 1.00000 11 TiB 2.1 TiB 2.1 TiB 8 KiB 12 GiB 8.8 TiB 19.28 0.55 169 up102 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 10 KiB 28 GiB 8.0 TiB 27.13 0.78 170 up 6 hdd 10.91399 1.00000 11 TiB 4.5 TiB 4.4 TiB 23 KiB 44 GiB 6.4 TiB 41.14 1.18 181 up 18 hdd 10.91399 1.00000 11 TiB 6.0 TiB 6.0 TiB 9 KiB 61 GiB 4.9 TiB 55.15 1.58 183 up 41 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 15 KiB 27 GiB 8.0 TiB 27.12 0.78 179 up 49 hdd 10.91399 1.00000 11 TiB 785 GiB 775 GiB 19 KiB 9.5 GiB 10 TiB 7.02 0.20 176 up 59 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 8 KiB 19 GiB 8.7 TiB 20.12 0.58 178 up 68 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 13 KiB 36 GiB 7.2 TiB 34.15 0.98 172 up 97 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 15 KiB 27 GiB 8.0 TiB 27.13 0.78 178 up105 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 13 KiB 36 GiB 7.2 TiB 34.14 0.98 179 up 8 hdd 10.91399 1.00000 11 TiB 6.6 TiB 6.5 TiB 13 KiB 55 GiB 4.3 TiB 60.40 1.73 181 up 17 hdd 10.91399 1.00000 11 TiB 4.6 TiB 4.5 TiB 14 KiB 52 GiB 6.3 TiB 42.03 1.20 177 up 27 hdd 10.91399 1.00000 11 TiB 783 GiB 774 GiB 8 KiB 9.4 GiB 10 TiB 7.01 0.20 176 up 38 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 15 KiB 19 GiB 8.7 TiB 20.11 0.58 174 up 57 hdd 10.91399 1.00000 11 TiB 2.3 TiB 2.3 TiB 12 KiB 26 GiB 8.6 TiB 21.01 0.60 177 up 67 hdd 10.91399 1.00000 11 TiB 4.5 TiB 4.4 TiB 9 KiB 44 GiB 6.4 TiB 41.14 1.18 176 up 93 hdd 10.91399 1.00000 11 TiB 3.1 TiB 3.0 TiB 7 KiB 35 GiB 7.9 TiB 28.01 0.80 171 up103 hdd 10.91399 1.00000 11 TiB 4.2 TiB 4.2 TiB 10 KiB 22 GiB 6.7 TiB 38.52 1.10 176 up three pools (deeproute-replica-hdd-pool, os-dsglczutvqsgowpz.rgw.buckets.data, and cephfs-replicated-pool ) are set to use the same device class hdd. I noticed that the *effective ratio* of the three pools is very different, not sure if it will affect the data rebalancing. [root@node01 smd]# ceph osd pool autoscale-statusPOOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK device_health_metrics 278.5M 3.0 48289G 0.0000 1.0 1 on False deeproute-replica-hdd-pool 12231M 3.0 785.8T 0.9901 50.0000 0.9901 1.0 4096 on False deeproute-replica-ssd-pool 12333G 3.0 48289G 0.9901 50.0000 0.9901 1.0 1024 on False .rgw.root 5831 3.0 48289G 0.0099 0.5000 0.0099 1.0 8 on False default.rgw.log 182 3.0 48289G 0.0000 1.0 32 on False default.rgw.control 0 3.0 48289G 0.0000 1.0 32 on False default.rgw.meta 0 3.0 48289G 0.0000 4.0 8 on False os-dsglczutvqsgowpz.rgw.control 0 3.0 16096G 0.1667 0.5000 0.1667 1.0 64 on False os-dsglczutvqsgowpz.rgw.meta 99.81k 3.0 16096G 0.1667 0.5000 0.1667 1.0 64 on False os-dsglczutvqsgowpz.rgw.buckets.index 59437M 3.0 16096G 0.1667 0.5000 0.1667 1.0 32 on False os-dsglczutvqsgowpz.rgw.buckets.non-ec 498.7M 3.0 16096G 0.1667 0.5000 0.1667 1.0 32 on False os-dsglczutvqsgowpz.rgw.log 1331M 3.0 16096G 0.1667 0.5000 0.1667 1.0 32 on False os-dsglczutvqsgowpz.rgw.buckets.data 143.1T 1.3333333730697632 785.8T 0.2429 0.5000 0.0099 1.0 32 on False cephfs-metadata 3161M 3.0 16096G 0.0006 4.0 32 on False cephfs-replicated-pool 21862G 3.0 785.8T 0.0815 1.0 32 on False .nfs 84599 3.0 48289G 0.0000 1.0 32 on False os-dsglczutvqsgowpz.rgw.otp 0 3.0 16096G 0.1667 0.5000 [root@node01 smd]# ceph df--- RAW STORAGE ---CLASS SIZE AVAIL USED RAW USED %RAW USEDhdd 786 TiB 545 TiB 240 TiB 240 TiB 30.60ssd 63 TiB 25 TiB 38 TiB 38 TiB 59.95TOTAL 849 TiB 571 TiB 278 TiB 278 TiB 32.77 --- POOLS ---POOL ID PGS STORED OBJECTS USED %USED MAX AVAILdevice_health_metrics 1 1 280 MiB 135 841 MiB 0.02 1.8 TiBdeeproute-replica-hdd-pool 11 4096 12 GiB 9.44k 36 GiB 0.02 69 TiBdeeproute-replica-ssd-pool 12 1024 12 TiB 3.43M 37 TiB 87.62 1.8 TiB.rgw.root 25 8 5.7 KiB 20 228 KiB 0 1.8 TiBdefault.rgw.log 26 32 182 B 2 24 KiB 0 1.8 TiBdefault.rgw.control 27 32 0 B 8 0 B 0 1.8 TiBdefault.rgw.meta 28 8 0 B 0 0 B 0 1.8 TiBos-dsglczutvqsgowpz.rgw.control 29 64 0 B 8 0 B 0 4.9 TiBos-dsglczutvqsgowpz.rgw.meta 30 64 100 KiB 456 4.3 MiB 0 4.9 TiBos-dsglczutvqsgowpz.rgw.buckets.index 31 32 58 GiB 10.40k 174 GiB 1.15 4.9 TiBos-dsglczutvqsgowpz.rgw.buckets.non-ec 32 32 500 MiB 247.78k 4.3 GiB 0.03 4.9 TiBos-dsglczutvqsgowpz.rgw.log 33 32 689 MiB 432 2.0 GiB 0.01 4.9 TiBos-dsglczutvqsgowpz.rgw.buckets.data 34 32 130 TiB 134.96M 174 TiB 45.69 155 TiBcephfs-metadata 35 32 3.1 GiB 1.67M 9.3 GiB 0.06 4.9 TiBcephfs-replicated-pool 36 32 21 TiB 8.25M 64 TiB 23.79 69 TiB.nfs 37 32 89 KiB 9 341 KiB 0 1.8 TiBos-dsglczutvqsgowpz.rgw.otp 38 8 0 B 0 0 B 0 4.9 TiB [root@node01 smd]# ceph -s cluster: id: f7f1c8ba-f793-436b-bb73-0964108a30c1 health: HEALTH_OK services: mon: 3 daemons, quorum a,b,c (age 7w) mgr: a(active, since 2w), standbys: b mds: 1/1 daemons up, 1 hot standby osd: 108 osds: 108 up (since 5h), 108 in (since 4M); 1 remapped pgs rgw: 9 daemons active (9 hosts, 1 zones) data: volumes: 1/1 healthy pools: 17 pools, 5561 pgs objects: 148.37M objects, 162 TiB usage: 276 TiB used, 572 TiB / 849 TiB avail pgs: 2786556/1119434807 objects misplaced (0.249%) 5559 active+clean 1 active+remapped+backfilling 1 active+clean+scrubbing+deep io: client: 212 MiB/s rd, 27 MiB/s wr, 4.04k op/s rd, 2.01k op/s wr recovery: 4.0 MiB/s, 4 objects/s progress: Global Recovery Event (2d) [===========================.] (remaining: 39m) So my question is, how can I adjust to make the osd data more evenly distributed? Thanks! Best Regards, Liang Zheng

1 year, 1 month

1
1
0 0

Generated signurl is accessible from restricted IPs in bucket policy

by Aggelos Toumasis

Hi there, We noticed after creating a signurl that the bucket resources were accessible from IPs that were originally restricted from accessing them (using a bucket policy). Using the s3cmd utility we confirmed that the Policy is correctly applied and you can access it only for the allowed IPs. Is this an expected behavior or do we miss something? Thanks!

1 year, 1 month

3
2
0 0

With Ceph Quincy, the "ceph" package does not include ceph-volume anymore

by Geert Kloosterman

Hi all, Until Ceph Pacific, installing just the "ceph" package was enough to get everything needed to deploy Ceph. However, with Quincy, ceph-volume was split off into its own package, and it is not automatically installed anymore. Here we can see it is not listed as a dependency: $ rpm -q --requires ceph binutils ceph-mds = 2:17.2.5-0.el8 ceph-mgr = 2:17.2.5-0.el8 ceph-mon = 2:17.2.5-0.el8 ceph-osd = 2:17.2.5-0.el8 rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(PayloadIsXz) <= 5.2-1 systemd Should I file a bug for this? Best regards, Geert Kloosterman

1 year, 1 month

4
4
0 0

RGW can't create bucket

by Kamil Madac

Hi, One of my customers had a correctly working RGW cluster with two zones in one zonegroup and since a few days ago users are not able to create buckets and are always getting Access denied. Working with existing buckets works (like listing/putting objects into existing bucket). The only operation which is not working is bucket creation. We also tried to create a new user, but the behavior is the same, and he is not able to create the bucket. We tried s3cmd, python script with boto library and also Dashboard as admin user. We are always getting Access Denied. Zones are in-sync. Has anyone experienced such behavior? Thanks in advance, here are some outputs: $ s3cmd -c .s3cfg_python_client mb s3://test ERROR: Access to bucket 'test' was denied ERROR: S3 error: 403 (AccessDenied) Zones are in-sync: Primary cluster: # radosgw-admin sync status realm 5429b434-6d43-4a18-8f19-a5720a89c621 (solargis-prod) zonegroup 00e4b3ff-1da8-4a86-9f52-4300c6d0f149 (solargis-prod-ba) zone 6067eec6-a930-45c7-af7d-a7ef2785a2d7 (solargis-prod-ba-dc) metadata sync no sync (zone is master) data sync source: e84fd242-dbae-466c-b4d9-545990590995 (solargis-prod-ba-hq) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source Secondary cluster: # radosgw-admin sync status realm 5429b434-6d43-4a18-8f19-a5720a89c621 (solargis-prod) zonegroup 00e4b3ff-1da8-4a86-9f52-4300c6d0f149 (solargis-prod-ba) zone e84fd242-dbae-466c-b4d9-545990590995 (solargis-prod-ba-hq) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 6067eec6-a930-45c7-af7d-a7ef2785a2d7 (solargis-prod-ba-dc) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source -- Kamil Madac <https://kmadac.github.io/>

1 year, 1 month

1
0
0 0

Unexpected ceph pool creation error with Ceph Quincy

by Geert Kloosterman

Hi all, I'm trying out Ceph Quincy (17.2.5) for the first time and I'm running into unexpected behavior of "ceph osd pool create". When not passing any pg_num and pgp_num values, I get the following error with Quincy: [root@gjk-ceph ~]# ceph osd pool create asdf Error ERANGE: 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1 I checked with Ceph Pacific (16.2.11) and there the extra arguments are not needed. I expected it would use osd_pool_default_pg_num and osd_pool_default_pgp_num as defined in my ceph.conf: [root@gjk-ceph ~]# ceph-conf -D | grep 'osd_pool_default_pg' osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 8 osd_pool_default_pgp_num = 8 At least, this is what appears to be used with Pacific. Is this an intended change of behavior? I could not find anything related in the release notes. Best regards, Geert Kloosterman

1 year, 1 month

2
5
0 0

Ceph performance problems

by Dominik Baack

Hi, we are currently testing out ways to increase Ceph performance because what we experience so far is very close to unusable. For the test cluster we utilizing 4 nodes with the following hardware data: Dual 200GBe Mellanox Ethernet 2x EPYC Rome 7302 16x 32GB 3200MHz ECC 9x 15.36TB Micron 9300 Pro For production this will be extended to all 8 nodes if it shows promising results. - Ceph was installed with cephadm. - MDS and ODS are located on the same nodes. - Mostly using stock config - Network performance tested with iperf3 seems fine, 26Gbits/s with -P4 on single port (details below). Close to 200Gbits with 10 parallel instances and servers. When testing a mounted CephFS on the working nodes in various configurations I only got <50MB/s for fuse mount and <270MB/s for kernel mounts. (dd command and output attached below) In addition ceph dashboard and our graphana monitoring reports packet loss on all relevant interfaces during load. Which does not occur during the normal iperf load tests or rsync/scp file transfer. Rados Bench shows performance around 2000MB/s which is not max performance of the SSDs but fine for us (details below). Why is the filesystem so slow compared to the individual components? Cheers Dominik Test details: ------------------------------------------------------------------------------------------------------ Some tests done on working nodes: Ceph mounted with ceph-fuse root@ml2ran10:/mnt/cephfs/backup# dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4,3 GB, 4,0 GiB) copied, 88,2933 s, 48,6 MB/s Ceph mounted with kernel driver: root@ml2ran06:/mnt/cephfs/backup# dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.0989 s, 267 MB/s Storage Node With fuse root@ml2rsn05:/mnt/ml2r_storage/backup# dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 53.9977 s, 79.5 MB/s Kernel mount: dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 17.6726 s, 243 MB/s _______________________________________________________ Iperf3 iperf3 --zerocopy -n 10240M -P4 -c ml2ran08s0 -p 4701 -i 15 -b 200000000000 Connecting to host ml2ran08s0, port 4701 [ 5] local 129.217.31.180 port 43958 connected to 129.217.31.218 port 4701 [ 7] local 129.217.31.180 port 43960 connected to 129.217.31.218 port 4701 [ 9] local 129.217.31.180 port 43962 connected to 129.217.31.218 port 4701 [ 11] local 129.217.31.180 port 43964 connected to 129.217.31.218 port 4701 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 632 KBytes [ 7] 0.00-3.21 sec 2.50 GBytes 6.70 Gbits/sec 0 522 KBytes [ 9] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 612 KBytes [ 11] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 430 KBytes [SUM] 0.00-3.21 sec 10.0 GBytes 26.8 Gbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender [ 5] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec receiver [ 7] 0.00-3.21 sec 2.50 GBytes 6.70 Gbits/sec 0 sender [ 7] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec receiver [ 9] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender [ 9] 0.00-3.21 sec 2.49 GBytes 6.67 Gbits/sec receiver [ 11] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender [ 11] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec receiver [SUM] 0.00-3.21 sec 10.0 GBytes 26.8 Gbits/sec 0 sender [SUM] 0.00-3.21 sec 9.98 GBytes 26.7 Gbits/sec receiver _________________________________________________________________ Rados Bench on storage node: # rados bench -p testbench 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_ml2rsn05_2829244 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 747 731 2923.84 2924 0.0178757 0.0216262 2 16 1506 1490 2979.71 3036 0.0308664 0.0213685 3 16 2267 2251 3000.99 3044 0.0259053 0.0212556 4 16 3058 3042 3041.62 3164 0.0227621 0.0209792 5 16 3850 3834 3066.8 3168 0.0130519 0.0208148 6 16 4625 4609 3072.26 3100 0.151371 0.0207904 7 16 5381 5365 3065.28 3024 0.0300368 0.0208345 8 16 6172 6156 3077.57 3164 0.0197728 0.0207714 9 16 6971 6955 3090.67 3196 0.0142751 0.0206786 10 14 7772 7758 3102.76 3212 0.0181034 0.020605 Total time run: 10.0179 Total writes made: 7772 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 3103.23 Stddev Bandwidth: 93.3676 Max bandwidth (MB/sec): 3212 Min bandwidth (MB/sec): 2924 Average IOPS: 775 Stddev IOPS: 23.3419 Max IOPS: 803 Min IOPS: 731 Average Latency(s): 0.020598 Stddev Latency(s): 0.00731743 Max latency(s): 0.151371 Min latency(s): 0.00966991 # rados bench -p testbench 10 seq hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 15 657 642 2567.32 2568 0.011104 0.022631 2 15 1244 1229 2456.9 2348 0.0115248 0.019485 3 16 1499 1483 1976.64 1016 0.00722983 0.0177887 4 15 1922 1907 1906.16 1696 0.0142242 0.0327382 5 15 2593 2578 2061.6 2684 0.011758 0.0301774 6 16 3142 3126 2083.23 2192 0.00915926 0.027478 7 16 3276 3260 1862.23 536 0.00824714 0.0267449 8 16 3606 3590 1794.43 1320 0.0118938 0.0350541 9 16 4293 4277 1900.32 2748 0.0301886 0.0330604 10 14 5003 4989 1995.04 2848 0.0389717 0.0314977 Total time run: 10.0227 Total reads made: 5003 Read size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 1996.67 Average IOPS: 499 Stddev IOPS: 202.3 Max IOPS: 712 Min IOPS: 134 Average Latency(s): 0.0314843 Max latency(s): 3.04463 Min latency(s): 0.00551523 # rados bench -p testbench 10 rand hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 15 15 0 0 0 - 0 1 15 680 665 2657.61 2660 0.00919807 0.0224833 2 15 1273 1258 2514.26 2372 0.00839656 0.0247125 3 16 1863 1847 2461.4 2356 0.00994467 0.0236565 4 16 2064 2048 2047.14 804 0.00809139 0.0223506 5 16 2064 2048 1637.79 0 - 0.0223506 6 16 2477 2461 1640.12 826 0.0286315 0.0383254 7 16 3102 3086 1762.89 2500 0.0267464 0.0349189 8 16 3513 3497 1748 1644 0.00890952 0.032269 9 16 3617 3601 1600 416 0.00626917 0.0316019 10 15 4014 3999 1599.18 1592 0.0461076 0.0393606 Total time run: 10.0481 Total reads made: 4014 Read size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 1597.91 Average IOPS: 399 Stddev IOPS: 239.089 Max IOPS: 665 Min IOPS: 0 Average Latency(s): 0.0394035 Max latency(s): 3.00962 Min latency(s): 0.00449537

1 year, 1 month

3
2
0 0

MDS host in OSD blacklist

by Frank Schilder

Hi all, we have an octopus v15.2.17 cluster and observe that one of our MDS hosts showed up in the OSD blacklist: # ceph osd blacklist ls 192.168.32.87:6801/3841823949 2023-03-22T10:08:02.589698+0100 192.168.32.87:6800/3841823949 2023-03-22T10:08:02.589698+0100 I see an MDS restart that might be related; see log snippets below. There are no clients running on this host, only OSDs and one MDS. What could be the reason for the blacklist entries? Thanks! Log snippets: Mar 21 10:07:54 ceph-23 journal: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f99e63d5700 time 2023-03-21T10:07:54.967936+0100 Mar 21 10:07:54 ceph-23 journal: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE) Mar 21 10:07:54 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:54 ceph-23 journal: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f99f4a25b92] Mar 21 10:07:54 ceph-23 journal: 2: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:54 ceph-23 journal: 3: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:54 ceph-23 journal: 4: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:54 ceph-23 journal: 5: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:54 ceph-23 journal: 6: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:54 ceph-23 journal: 7: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:54 ceph-23 journal: 8: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:54 ceph-23 journal: 9: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:54 ceph-23 journal: 10: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:54 ceph-23 journal: *** Caught signal (Aborted) ** Mar 21 10:07:54 ceph-23 journal: in thread 7f99e63d5700 thread_name:MR_Finisher Mar 21 10:07:54 ceph-23 journal: 2023-03-21T10:07:54.980+0100 7f99e63d5700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f99e63d5700 time 2023-03-21T10:07:54.967936+0100 Mar 21 10:07:54 ceph-23 journal: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE) Mar 21 10:07:54 ceph-23 journal: Mar 21 10:07:54 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:54 ceph-23 journal: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f99f4a25b92] Mar 21 10:07:54 ceph-23 journal: 2: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:54 ceph-23 journal: 3: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:54 ceph-23 journal: 4: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:54 ceph-23 journal: 5: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:54 ceph-23 journal: 6: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:54 ceph-23 journal: 7: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:54 ceph-23 journal: 8: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:54 ceph-23 journal: 9: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:54 ceph-23 journal: 10: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:54 ceph-23 journal: Mar 21 10:07:54 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:54 ceph-23 journal: 1: (()+0x12ce0) [0x7f99f3605ce0] Mar 21 10:07:54 ceph-23 journal: 2: (gsignal()+0x10f) [0x7f99f2062a9f] Mar 21 10:07:54 ceph-23 journal: 3: (abort()+0x127) [0x7f99f2035e05] Mar 21 10:07:54 ceph-23 journal: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f99f4a25be3] Mar 21 10:07:54 ceph-23 journal: 5: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:54 ceph-23 journal: 6: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:54 ceph-23 journal: 7: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:54 ceph-23 journal: 8: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:54 ceph-23 journal: 9: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:54 ceph-23 journal: 10: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:54 ceph-23 journal: 11: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:54 ceph-23 journal: 12: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:54 ceph-23 journal: 13: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:54 ceph-23 journal: 2023-03-21T10:07:54.982+0100 7f99e63d5700 -1 *** Caught signal (Aborted) ** Mar 21 10:07:54 ceph-23 journal: in thread 7f99e63d5700 thread_name:MR_Finisher Mar 21 10:07:54 ceph-23 journal: Mar 21 10:07:54 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:54 ceph-23 journal: 1: (()+0x12ce0) [0x7f99f3605ce0] Mar 21 10:07:54 ceph-23 journal: 2: (gsignal()+0x10f) [0x7f99f2062a9f] Mar 21 10:07:54 ceph-23 journal: 3: (abort()+0x127) [0x7f99f2035e05] Mar 21 10:07:54 ceph-23 journal: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f99f4a25be3] Mar 21 10:07:54 ceph-23 journal: 5: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:54 ceph-23 journal: 6: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:54 ceph-23 journal: 7: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:54 ceph-23 journal: 8: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:54 ceph-23 journal: 9: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:54 ceph-23 journal: 10: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:54 ceph-23 journal: 11: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:54 ceph-23 journal: 12: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:54 ceph-23 journal: 13: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:54 ceph-23 journal: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 21 10:07:54 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: -1> 2023-03-21T10:07:54.980+0100 7f99e63d5700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f99e63d5700 time 2023-03-21T10:07:54.967936+0100 Mar 21 10:07:55 ceph-23 journal: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE) Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:55 ceph-23 journal: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f99f4a25b92] Mar 21 10:07:55 ceph-23 journal: 2: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:55 ceph-23 journal: 3: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:55 ceph-23 journal: 4: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:55 ceph-23 journal: 5: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:55 ceph-23 journal: 6: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:55 ceph-23 journal: 7: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:55 ceph-23 journal: 8: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:55 ceph-23 journal: 9: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:55 ceph-23 journal: 10: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: 0> 2023-03-21T10:07:54.982+0100 7f99e63d5700 -1 *** Caught signal (Aborted) ** Mar 21 10:07:55 ceph-23 journal: in thread 7f99e63d5700 thread_name:MR_Finisher Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:55 ceph-23 journal: 1: (()+0x12ce0) [0x7f99f3605ce0] Mar 21 10:07:55 ceph-23 journal: 2: (gsignal()+0x10f) [0x7f99f2062a9f] Mar 21 10:07:55 ceph-23 journal: 3: (abort()+0x127) [0x7f99f2035e05] Mar 21 10:07:55 ceph-23 journal: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f99f4a25be3] Mar 21 10:07:55 ceph-23 journal: 5: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:55 ceph-23 journal: 6: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:55 ceph-23 journal: 7: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:55 ceph-23 journal: 8: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:55 ceph-23 journal: 9: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:55 ceph-23 journal: 10: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:55 ceph-23 journal: 11: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:55 ceph-23 journal: 12: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:55 ceph-23 journal: 13: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:55 ceph-23 journal: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: -9999> 2023-03-21T10:07:54.980+0100 7f99e63d5700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f99e63d5700 time 2023-03-21T10:07:54.967936+0100 Mar 21 10:07:55 ceph-23 journal: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE) Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:55 ceph-23 journal: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f99f4a25b92] Mar 21 10:07:55 ceph-23 journal: 2: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:55 ceph-23 journal: 3: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:55 ceph-23 journal: 4: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:55 ceph-23 journal: 5: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:55 ceph-23 journal: 6: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:55 ceph-23 journal: 7: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:55 ceph-23 journal: 8: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:55 ceph-23 journal: 9: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:55 ceph-23 journal: 10: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: -9998> 2023-03-21T10:07:54.982+0100 7f99e63d5700 -1 *** Caught signal (Aborted) ** Mar 21 10:07:55 ceph-23 journal: in thread 7f99e63d5700 thread_name:MR_Finisher Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:55 ceph-23 journal: 1: (()+0x12ce0) [0x7f99f3605ce0] Mar 21 10:07:55 ceph-23 journal: 2: (gsignal()+0x10f) [0x7f99f2062a9f] Mar 21 10:07:55 ceph-23 journal: 3: (abort()+0x127) [0x7f99f2035e05] Mar 21 10:07:55 ceph-23 journal: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f99f4a25be3] Mar 21 10:07:55 ceph-23 journal: 5: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:55 ceph-23 journal: 6: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:55 ceph-23 journal: 7: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:55 ceph-23 journal: 8: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:55 ceph-23 journal: 9: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:55 ceph-23 journal: 10: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:55 ceph-23 journal: 11: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:55 ceph-23 journal: 12: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:55 ceph-23 journal: 13: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:55 ceph-23 journal: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: reraise_fatal: default handler for signal 6 didn't terminate the process? Mar 21 10:07:58 ceph-23 dockerd-current: time="2023-03-21T10:07:58.119559277+01:00" level=warning msg="040c1e98a0669204e0e98bdbcdde893f8acf63444f3827358e663a13a2869478 cleanup: failed to unmount secrets: invalid argument" Mar 21 10:07:58 ceph-23 kernel: overlayfs: upperdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior. Mar 21 10:07:58 ceph-23 kernel: overlayfs: workdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior. Mar 21 10:07:58 ceph-23 journal: 118 get_config /opt/ceph-container/bin/config.static.sh Mar 21 10:07:58 ceph-23 journal: 5 start_mds /opt/ceph-container/bin/start_mds.sh Mar 21 10:07:58 ceph-23 journal: 120 main /opt/ceph-container/bin/entrypoint.sh Mar 21 10:07:58 ceph-23 journal: 2023-03-21 10:07:58 /opt/ceph-container/bin/entrypoint.sh: static: does not generate config Mar 21 10:07:58 ceph-23 journal: 58 start_mds /opt/ceph-container/bin/start_mds.sh Mar 21 10:07:58 ceph-23 journal: 120 main /opt/ceph-container/bin/entrypoint.sh Mar 21 10:07:58 ceph-23 journal: 2023-03-21 10:07:58 /opt/ceph-container/bin/entrypoint.sh: SUCCESS Mar 21 10:07:58 ceph-23 journal: starting mds.ceph-23 at ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 year, 1 month

2
3
0 0

s3 compatible interface

by Marc

Anyone know of a s3 compatible interface that I can just run, and reads/writes files from a local file system and not from object storage?

1 year, 1 month

13
18
0 0

S3 notification for backup

by Olivier Audry

Hello I would like to know if using the bucket notification system to http for backuping S3 bucket is a good move or not. Is someone already did it ? I got around 100to of little documents to backup and archive for legal purpose and dr each day. Many thanks for your help. oau

1 year, 1 month

1
0
0 0

Moving From BlueJeans to Jitsi for Ceph meetings

by Mike Perez

Hi everyone, We have been using BlueJeans to meet and record some of our meetings that later get posted to our YouTube channel. Unfortunately, we have to figure out a new meeting platform due to Red Hat discontinuing BlueJeans by the end of this month. Google Meets is an option, but some users in other countries have trouble using Google's services. For some meetings, we have tried out Jitsi, and it works well, meets our requirements, and is free. Does anyone else have suggestions for another free meeting platform that provides recording capabilities? -- Mike Perez

1 year, 1 month

6
6
0 0

2024

2023

2022

2021

2020

2019

ceph-users March 2023