April 2023 - ceph-users - lists.ceph.io

by Yaarit Hatuka

Hi everyone, We are excited to share with you the latest statistics from our Ceph public telemetry dashboards <https://telemetry-public.ceph.com/>. With almost 2,500 clusters reporting nearly 1 exabyte of total storage capacity, we can see a steady growth rate <https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?viewPanel=19&orgId=…> since the telemetry service was launched. One of the things telemetry helps us to understand is version adoption rate. See, for example, the trend of Quincy <https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?viewPanel=28&orgId=…> deployments in the community. We can also learn that the average cluster size grows over time <https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?viewPanel=33&orgId=1>. Ceph telemetry is on an opt-in basis. You can opt-in with: `ceph telemetry on` Learn more here <https://docs.ceph.com/en/latest/mgr/telemetry/#enabling-telemetry>. Help us cross the exabyte mark by opting-in today! Learn more about the latest developments around Telemetry <https://sched.co/1JKZ2> at the upcoming Cephalocon. Thank you for your contribution, Yaarit [image: ceph_telemetry_dashboards.jpg]

1 year

2
3
0 0

RGW Lua - cancel request

by Ondřej Kukla

Hello, Lately I’ve been playing with Lua scripting on top of RGW. I would like to implement a request blocking based on bucket name -> when there is a dot in a bucket name return error code and a message that this name is invalid. Here is the code I was able to came up with. if string.find(Request.HTTP.URI, '%.') then Request.Response.HTTPStatusCode = 400 Request.Response.HTTPStatus = “InvalidBucketName" Request.Response.Message = “Dots in bucket name are not allowed." end This works fine, but the request for creating a bucket would be processed and the bucket will be created. I thought about a dirty workaround with setting the Request.Bucket.Name to a bucket that already exists but it seems that this field is not writable in Quincy. Is there a way to block the request from processing? Any help is much appreciated. Kind regards, Ondrej

1 year

3
3
0 0

backing up CephFS

by Angelo Höngens

How do you guys backup CephFS? (if at all?) I'm building 2 ceph clusters, a primary one and a backup one, and I'm looking into CephFS as the primary store for research files. CephFS mirroring seems a very fast and efficient way to copy data to the backup location, and it has the benefit of the files on the backup location being fully in a ready-to-use state instead of some binary proprietary archive. But I am wondering how to do 'ransomware protection' in this setup. I can't believe I'm the only one that wants to secure my data ;) I'm reading up on snapshots and mirroring, and that's great to protect from user error. I could schedule snapshots on the primary cluster, and they would automatically get synced to the backup cluster. But a user can still delete all snapshots on the source side, right? And you need to create a ceph user on the backup cluster, and import that on the primary cluster. That means that if a hacker has those credentials, he could also delete the data on the backup cluster? Or is there some 'append-only' mode for immutability? Another option I'm looking into is restic. Restic looks like a cool tool, but it does not support s3 object locks yet. See the discussion here [1]. I should be able to get immutability working with the restic-rest backend according to the developer. But I have my worries that running restic to sync up an 800TB filesystem with millions of files will be.. worrysome ;) Anyone using restic in production? Thanks again for your input! Angelo. [1] https://github.com/restic/restic/issues/3195

1 year

4
3
0 0

Deep-scrub much slower than HDD speed

by Niklas Hambüchen

I observed that on an otherwise idle cluster, scrubbing cannot fully utilise the speed of my HDDs. `iostat` shows only 8-10 MB/s per disk, instead of the ~100 MB/s most HDDs can easily deliver. Changing scrubbing settings does not help (see below). Environment: * 6 active+clean+scrubbing+deep * Ceph version 16.2.7. * BlueStore * My cluster has many objects small objects ("402.32M objects, 38 TiB" from "ceph status") due to small files (4 - 32 KiB) on CephFS. `iostat -x 5` with default settings: Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util dm-0 198.60 6878.40 0.00 0.00 12.78 34.63 51.80 2612.80 0.00 0.00 14.82 50.44 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.30 91.24 dm-1 0.80 3.20 0.00 0.00 11.50 4.00 52.60 2582.40 0.00 0.00 13.69 49.10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.73 3.78 dm-10 11.20 71.20 0.00 0.00 0.09 6.36 145.80 583.20 0.00 0.00 0.14 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 2.62 dm-11 192.60 6737.60 0.00 0.00 10.74 34.98 34.80 1684.80 0.00 0.00 11.47 48.41 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.47 91.40 dm-12 245.40 10194.40 0.00 0.00 9.43 41.54 21.20 575.20 0.00 0.00 3.94 27.13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.40 87.92 dm-13 30.80 1772.80 0.00 0.00 11.61 57.56 78.80 4507.20 0.00 0.00 19.78 57.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.92 9.54 dm-14 3.20 24.80 0.00 0.00 0.12 7.75 125.20 500.80 0.00 0.00 0.12 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.18 dm-15 2.80 19.20 0.00 0.00 0.14 6.86 105.40 421.60 0.00 0.00 0.05 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.76 dm-16 0.80 6.40 0.00 0.00 0.00 8.00 111.00 444.00 0.00 0.00 0.10 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.82 dm-17 10.80 76.80 0.00 0.00 0.09 7.11 151.40 605.60 0.00 0.00 0.08 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.92 dm-18 10.20 67.20 0.00 0.00 0.08 6.59 115.60 462.40 0.00 0.00 0.04 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.16 dm-19 10.20 56.80 0.00 0.00 0.10 5.57 109.00 436.00 0.00 0.00 0.07 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.34 dm-2 4.80 435.20 0.00 0.00 0.12 90.67 751.80 6292.80 0.00 0.00 0.07 8.37 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.05 16.14 dm-20 0.40 2.40 0.00 0.00 0.00 6.00 265.00 2459.20 0.00 0.00 0.10 9.28 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 5.36 dm-21 191.00 6105.60 0.00 0.00 6.34 31.97 67.80 3748.00 0.00 0.00 19.56 55.28 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.54 42.22 dm-3 1.00 8.80 0.00 0.00 0.00 8.80 91.00 364.00 0.00 0.00 0.04 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.54 dm-4 167.60 4973.60 0.00 0.00 10.15 29.68 49.20 2511.20 0.00 0.00 11.39 51.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.26 89.18 dm-5 11.20 73.60 0.00 0.00 0.12 6.57 124.40 497.60 0.00 0.00 0.07 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.16 dm-6 27.20 1644.80 0.00 0.00 12.22 60.47 57.20 3316.80 0.00 0.00 15.93 57.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.24 6.78 dm-7 217.40 8032.80 0.00 0.00 12.04 36.95 64.40 3654.40 0.00 0.00 23.69 56.75 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 4.14 97.28 dm-8 10.80 70.40 0.00 0.00 0.15 6.52 111.80 447.20 0.00 0.00 0.04 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.32 dm-9 1.60 8.00 0.00 0.00 13.25 5.00 46.60 2563.20 0.00 0.00 9.08 55.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.44 3.70 md127 1.60 107.20 0.00 0.00 0.00 67.00 142.00 1856.00 0.00 0.00 0.05 13.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.80 nvme0n1 42.00 772.00 0.00 0.00 0.10 18.38 1001.00 10503.50 451.20 31.07 0.03 10.49 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.04 25.20 nvme1n1 37.20 248.00 0.00 0.00 0.09 6.67 609.80 6723.50 369.00 37.70 0.04 11.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 15.80 sda 0.80 3.20 0.00 0.00 11.50 4.00 33.60 2582.40 19.00 36.12 8.43 76.86 0.00 0.00 0.00 0.00 0.00 0.00 3.20 0.25 0.29 3.78 sdb 117.80 10195.20 127.80 52.04 11.53 86.55 18.80 575.20 2.40 11.32 3.93 30.60 0.00 0.00 0.00 0.00 0.00 0.00 2.80 3.93 1.44 87.90 sdc 4.60 1644.80 22.60 83.09 11.74 357.57 23.60 3316.80 33.60 58.74 8.53 140.54 0.00 0.00 0.00 0.00 0.00 0.00 1.80 4.56 0.26 6.74 sdd 109.40 4975.20 58.40 34.80 12.19 45.48 22.00 2511.20 27.20 55.28 5.75 114.15 0.00 0.00 0.00 0.00 0.00 0.00 2.80 6.07 1.48 89.16 sde 115.40 6563.20 77.00 40.02 12.21 56.87 26.20 1684.80 8.60 24.71 7.82 64.31 0.00 0.00 0.00 0.00 0.00 0.00 1.80 6.78 1.63 91.36 sdf 6.40 1772.80 24.40 79.22 14.84 277.00 39.20 4507.20 39.60 50.25 13.93 114.98 0.00 0.00 0.00 0.00 0.00 0.00 3.20 2.25 0.65 9.54 sdg 121.60 8033.60 95.80 44.07 12.39 66.07 30.60 3654.40 33.80 52.48 12.47 119.42 0.00 0.00 0.00 0.00 0.00 0.00 2.60 8.77 1.91 97.24 sdh 122.00 6105.60 69.00 36.13 5.47 50.05 33.00 3748.00 34.80 51.33 16.94 113.58 0.00 0.00 0.00 0.00 0.00 0.00 2.20 5.00 1.24 42.20 sdi 117.00 6856.80 81.60 41.09 12.25 58.61 32.00 2612.80 19.80 38.22 10.80 81.65 0.00 0.00 0.00 0.00 0.00 0.00 2.60 8.46 1.80 91.18 sdj 1.60 8.00 0.00 0.00 13.12 5.00 31.20 2563.20 15.40 33.05 5.06 82.15 0.00 0.00 0.00 0.00 0.00 0.00 2.00 1.70 0.18 3.70 With settings osd_deep_scrub_stride = 4194304 osd_scrub_load_threshold = 20 osd_scrub_chunk_min = 15 osd_scrub_chunk_max = 75 osd_max_scrubs = 3 I get a slight improvement only: Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util dm-0 400.60 14686.40 0.00 0.00 19.93 36.66 25.20 1197.60 0.00 0.00 27.44 47.52 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 8.67 91.32 dm-1 362.60 10583.20 0.00 0.00 18.58 29.19 30.40 1742.40 0.00 0.00 23.32 57.32 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.45 98.14 dm-10 6.40 64.00 0.00 0.00 0.03 10.00 37.60 150.40 0.00 0.00 0.06 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.02 dm-11 76.60 1939.20 0.00 0.00 10.61 25.32 35.20 1885.60 0.00 0.00 22.32 53.57 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.60 29.06 dm-12 93.00 4178.40 0.00 0.00 16.04 44.93 4.40 96.00 0.00 0.00 0.68 21.82 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.50 19.04 dm-13 376.80 12716.00 0.00 0.00 15.90 33.75 27.20 1444.80 0.00 0.00 23.76 53.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 6.64 97.76 dm-14 11.80 312.80 0.00 0.00 0.12 26.51 33.40 133.60 0.00 0.00 0.00 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.96 dm-15 7.00 430.40 0.00 0.00 0.17 61.49 46.20 184.80 0.00 0.00 0.06 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.94 dm-16 11.80 229.60 0.00 0.00 0.10 19.46 49.20 196.80 0.00 0.00 0.09 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.20 dm-17 4.00 39.20 0.00 0.00 0.10 9.80 29.60 118.40 0.00 0.00 0.00 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.72 dm-18 9.00 94.40 0.00 0.00 0.07 10.49 37.20 148.80 0.00 0.00 0.30 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.92 dm-19 7.40 70.40 0.00 0.00 0.11 9.51 56.80 227.20 0.00 0.00 0.02 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.04 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 62.00 482.40 0.00 0.00 0.07 7.78 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.16 dm-20 12.40 2375.20 0.00 0.00 0.23 191.55 314.20 1524.80 0.00 0.00 0.03 4.85 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 7.90 dm-21 321.00 9468.00 0.00 0.00 7.86 29.50 31.20 1630.40 0.00 0.00 11.69 52.26 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.89 42.84 dm-3 12.40 125.60 0.00 0.00 0.08 10.13 37.00 148.00 0.00 0.00 0.02 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.84 dm-4 374.00 13150.40 0.00 0.00 18.31 35.16 23.60 1100.80 0.00 0.00 12.42 46.64 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.14 96.70 dm-5 8.00 84.00 0.00 0.00 0.10 10.50 38.00 152.00 0.00 0.00 0.03 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 dm-6 201.60 6619.20 0.00 0.00 11.56 32.83 3.40 108.80 0.00 0.00 2.29 32.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.34 58.24 dm-7 414.40 14476.00 0.00 0.00 21.99 34.93 10.80 235.20 0.00 0.00 5.19 21.78 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 9.17 98.00 dm-8 0.60 14.40 0.00 0.00 0.00 24.00 40.80 163.20 0.00 0.00 0.04 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.70 dm-9 478.00 17300.80 0.00 0.00 23.06 36.19 4.40 92.00 0.00 0.00 5.00 20.91 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11.05 98.00 md127 5.00 64.80 0.00 0.00 0.04 12.96 133.80 2014.40 0.00 0.00 0.08 15.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.32 nvme0n1 33.60 822.40 0.00 0.00 0.09 24.48 252.40 3388.10 139.40 35.58 0.04 13.42 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 7.24 nvme1n1 62.60 3082.40 0.00 0.00 0.10 49.24 427.40 4272.90 177.20 29.31 0.03 10.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 13.70 sda 236.20 10613.60 126.20 34.82 15.19 44.93 11.80 1742.40 18.60 61.18 13.83 147.66 0.00 0.00 0.00 0.00 0.00 0.00 1.40 20.29 3.78 98.14 sdb 35.80 4166.40 56.80 61.34 10.03 116.38 4.20 96.00 0.20 4.55 0.81 22.86 0.00 0.00 0.00 0.00 0.00 0.00 0.80 0.25 0.36 19.04 sdc 118.20 6612.80 83.00 41.25 9.15 55.95 3.00 108.80 0.40 11.76 2.07 36.27 0.00 0.00 0.00 0.00 0.00 0.00 0.60 3.67 1.09 58.20 sdd 207.00 13152.00 166.60 44.59 13.92 63.54 15.80 1100.80 7.80 33.05 6.90 69.67 0.00 0.00 0.00 0.00 0.00 0.00 1.80 9.67 3.01 96.66 sde 55.80 1939.20 20.80 27.15 9.06 34.75 17.40 1885.60 17.80 50.57 7.51 108.37 0.00 0.00 0.00 0.00 0.00 0.00 1.80 1.44 0.64 29.04 sdf 221.40 12720.00 155.20 41.21 14.34 57.45 14.60 1444.80 12.60 46.32 10.51 98.96 0.00 0.00 0.00 0.00 0.00 0.00 1.40 11.71 3.35 97.70 sdg 234.60 14490.40 179.80 43.39 15.66 61.77 9.40 235.20 1.20 11.32 5.36 25.02 0.00 0.00 0.00 0.00 0.00 0.00 2.00 11.10 3.75 97.98 sdh 212.20 9470.40 109.20 33.98 4.05 44.63 19.40 1630.40 11.80 37.82 6.41 84.04 0.00 0.00 0.00 0.00 0.00 0.00 2.00 4.10 0.99 42.78 sdi 214.00 14684.00 186.60 46.58 14.46 68.62 10.00 1197.60 15.20 60.32 11.14 119.76 0.00 0.00 0.00 0.00 0.00 0.00 2.40 13.67 3.24 91.28 sdj 250.60 17296.80 227.00 47.53 18.76 69.02 3.60 92.00 0.80 18.18 5.72 25.56 0.00 0.00 0.00 0.00 0.00 0.00 0.80 16.25 4.73 98.00 `osd_max_scrubs` creates more reads per second (~120 -> ~220), but does not proportionately increase read throughput (as is expected on a spinning disk). Overall, this looks like scrub operations are seek-bound. Looking at `strace -fyp 640105 -e io_submit` output, for lines that involve my HDD `/dev/dm-12`: io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd55baa3000, iov_len=4096}], aio_offset=1654046691328}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f378000, iov_len=24576}], aio_offset=1654046666752}]) = 2 io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f262000, iov_len=4096}], aio_offset=1934563819520}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f28b000, iov_len=4096}], aio_offset=1934563823616}]) = 2 io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f3c7000, iov_len=28672}], aio_offset=2871307956224}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f2d2000, iov_len=4096}], aio_offset=2871307984896}]) = 2 io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f34a000, iov_len=8192}], aio_offset=4056494669824}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f35f000, iov_len=4096}], aio_offset=4056494678016}]) = 2 io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f3e3000, iov_len=53248}], aio_offset=1233895051264}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f0c6000, iov_len=4096}], aio_offset=1233895104512}]) = 2 io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f1e2000, iov_len=4096}], aio_offset=2155792314368}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f303000, iov_len=24576}], aio_offset=2155792289792}]) = 2 This looks like it's reading the HDD all over the place, with small reads (`aio_offset`, `iov_len`). Does Ceph scrubbing do a HDD seek for every object? I had imagined that scrubbing might be able to read linearly through the disk (perhaps skipping larger gaps). Is that wrong? Thanks!

1 year

7
16
0 0

Enable LUKS encryption on a snapshot created from unencrypted image

by Will Gorman

Is there a way to enable the LUKS encryption format on a snapshot that was created from an unencrypted image without losing data? I've seen in https://docs.ceph.com/en/quincy/rbd/rbd-encryption/ that "Any data written to the image prior to its format may become unreadable, though it may still occupy storage resources." and observed that to be the case when running `encryption format` on an image that already has data in it. However is there any way to take a snapshot of an unencrypted image and enable encryption on the snapshot (or even on a new image cloned from the snapshot?)

1 year

2
1
1 0

cephfs - max snapshot limit?

by Tobias Hachmer

Hello, we are running a 3-node ceph cluster with version 17.2.6. For CephFS snapshots we have configured the following snap schedule with retention: /PATH 2h 72h15d6m But we observed that max 50 snapshot are preserved. If a new snapshot is created the oldest 51st is deleted. Is there a limit for maximum cephfs snapshots or maybe this is a bug? I have found the setting "mds_max_snaps_per_dir" which is 100 by default but I think this is not related to my problem? Thanks, Tobias

1 year

4
9
0 0

Return code -116 when starting MDS scrub

by Thomas Widhalm

Hi, I followed the steps to repair journal and MDS I found here in the list. I hit a bug that stopped my MDS to start so I took the long way with reading the data. Everything went fine and I can even mount one of my CephFS now. That's a big relieve. But when I start scrub, I just get return code -116 and no scrub is initiatet. I didn't find that code in the docs. Can you help me? [ceph: root@ceph06 /]# ceph tell mds.mds01.ceph06.huavsw scrub start recursive 2023-04-29T10:46:36.926+0000 7ff676ff5700 0 client.79389355 ms_handle_reset on v2:192.168.23.66:6800/1133836262 2023-04-29T10:46:36.953+0000 7ff676ff5700 0 client.79389361 ms_handle_reset on v2:192.168.23.66:6800/1133836262 { "return_code": -116 } (I get the same error, no matter what kind of scrub I start) -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

1 year

2
2
0 0

import OSD after host OS reinstallation

by Tony Liu

Hi, The cluster is with Pacific and deployed by cephadm on container. The case is to import OSDs after host OS reinstallation. All OSDs are SSD who has DB/WAL and data together. Did some research, but not able to find a working solution. Wondering if anyone has experiences in this? What needs to be done before host OS reinstallation and what's after? Thanks! Tony

1 year

2
5
0 0

Block RGW request using Lua

by ondrej＠kuuk.la

Hello everyone, I've started playing with Lua scripting and would like to ask If anyone knows about a way to drop or close user request on the prerequest context. I would like to block creating buckets with dots in the name, but the use-case could be blocking certain operations, etc. I was able to come up with some like this if string.find(Request.HTTP.URI, '%.') then Request.Response.HTTPStatusCode = 400 Request.Response.HTTPStatus = "InvalidBucketName" Request.Response.Message = "Dots are not allowed." end This works fine, but the bucket is created which is something that I don't want to do. As a dirty workaround, I've thought about changing the bucket name here to an already existing bucket, but the Request.Bucket.Name = "taken" doesn't seem to work as the log gives me an error "attempt to index a nil value (field 'Bucket')". Any help is much appreciated.

1 year

1
0
0 0

For suggestions and best practices on expanding Ceph cluster and removing old nodes

by huxiaoyu＠horebdata.cn

Dear Ceph folks, I would like to listen to your advice on the following topic: We have a 6-node Ceph cluster (for RGW usage only ) running on Luminous 12.2.12, and now will add 10 new nodes. Our plan is to phase out the old 6 nodes, and run RGW Ceph cluster with the new 10 nodes on Nautilus version。 I can think of two ways to achieve the above goal. The first method would be: 1) Upgrade the current 6-node cluster from Luminous 12.2.12 to Nautilus 14.2.22; 2) Expand the cluster with the 10 new nodes, and then re-balance; 3) After rebalance completes, remove the 6 old nodes from the cluster The second method would get rid of the procedure to upgrade the old 6-node from Luminous to Nautilus, because those 6 nodes will be phased out anyway, but then we have to deal with a hybrid cluster with 6-node on Luminous 12.2.12, and 10-node on Nautilus, and after re-balancing, we can remove the 6 old nodes from the cluster. Any suggestions, advice, or best practice would be highly appreciated. best regards, Samuel huxiaoyu(a)horebdata.cn

1 year

7
8
0 0

2024

2023

2022

2021

2020

2019

ceph-users April 2023