- ceph-users - lists.ceph.io

Re: Issue about rbd image(disable feature journaling failed)
by Jason Dillaman 24 Mar '21

24 Mar '21

"rbd status" will tell you which IP addresses have the image open. "rbd lock ls" will show you who owns the lock. On Wed, Mar 24, 2021 at 9:07 AM Edgelong Voodu <1070443499cs(a)gmail.com> wrote: > > Hi, > As info mentioned in the log file > There still a lock owner alive. The question is how to find out the alive owner and what is the root cause about this. Why cannot acquire lock to the owner? > Thank you so much > > Jason Dillaman <jdillama(a)redhat.com>于2021年3月24日周三20:55写道： >> >> It sounds like this is a non-primary mirrored image, which means it's >> read-only and cannot be modified. A quick "rbd info" will tell you the >> mirror state. Instead, you would need to force-promote it to primary >> via "rbd mirror image promote --force" before attempting to modify the >> image. >> >> On Wed, Mar 24, 2021 at 3:44 AM Edgelong Voodu <1070443499cs(a)gmail.com> wrote: >> > >> > hi , dillaman: >> > i was confused nowadays, when i execute the rbd feature disable <image> journaling, >> > the result of this often shows me : >> > rbd: failed to update image features: (30) Read-only file system >> > i use the rbd-mirror for replicate data between two clouster. i thought the rbd-mirror could be the reason. but i have no idea about it. could you give some advise ? >> > some log when execute rbd command also send you. >> >> >> >> -- >> Jason >> > -- > 发自移动版 Gmail -- Jason

1 0

Re: Issue about rbd image(disable feature journaling failed)
by Jason Dillaman 24 Mar '21

24 Mar '21

That means you have another librbd client that "owns" the image that you will need to shut down before you can make that change. On Wed, Mar 24, 2021 at 9:01 AM Edgelong Voodu <1070443499cs(a)gmail.com> wrote: > > hi， > Thank you for the clarification. > It’s already promoted and not mirrored ,a primary image > I have run rbd —debug-rbd=30 and collect a log file > Which shows locker owner still alive and unable get the lock, return -EAGAIN > I’ll send you log later > > Thank you so much > > > Jason Dillaman <jdillama(a)redhat.com>于2021年3月24日周三20:55写道： >> >> It sounds like this is a non-primary mirrored image, which means it's >> read-only and cannot be modified. A quick "rbd info" will tell you the >> mirror state. Instead, you would need to force-promote it to primary >> via "rbd mirror image promote --force" before attempting to modify the >> image. >> >> On Wed, Mar 24, 2021 at 3:44 AM Edgelong Voodu <1070443499cs(a)gmail.com> wrote: >> > >> > hi , dillaman: >> > i was confused nowadays, when i execute the rbd feature disable <image> journaling, >> > the result of this often shows me : >> > rbd: failed to update image features: (30) Read-only file system >> > i use the rbd-mirror for replicate data between two clouster. i thought the rbd-mirror could be the reason. but i have no idea about it. could you give some advise ? >> > some log when execute rbd command also send you. >> >> >> >> -- >> Jason >> > -- > 发自移动版 Gmail -- Jason

1 0

Re: Issue about rbd image(disable feature journaling failed)
by Jason Dillaman 24 Mar '21

24 Mar '21

It sounds like this is a non-primary mirrored image, which means it's read-only and cannot be modified. A quick "rbd info" will tell you the mirror state. Instead, you would need to force-promote it to primary via "rbd mirror image promote --force" before attempting to modify the image. On Wed, Mar 24, 2021 at 3:44 AM Edgelong Voodu <1070443499cs(a)gmail.com> wrote: > > hi , dillaman: > i was confused nowadays, when i execute the rbd feature disable <image> journaling, > the result of this often shows me : > rbd: failed to update image features: (30) Read-only file system > i use the rbd-mirror for replicate data between two clouster. i thought the rbd-mirror could be the reason. but i have no idea about it. could you give some advise ? > some log when execute rbd command also send you. -- Jason

1 0

should I increase the amount of PGs?
by Boris Behrens 24 Mar '21

24 Mar '21

Hello people, I am still struggeling with the balancer (https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html) Now I've read some more and might think that I do not have enough PGs. Currently I have 84OSDs and 1024PGs for the main pool (3008 total). I have the autoscaler enabled, but I doesn't tell me to increase the PGs. What do you think? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

3 19

fixing future rctimes
by Dan van der Ster 23 Mar '21

23 Mar '21

Hi all, We have a few subdirs with an rctime in the future. # getfattr -n ceph.dir.rctime session # file: session ceph.dir.rctime="2576387188.090" I can't find any subdir or item in that directory with that rctime, so I presume that there was previously a file and that rctime cannot go backwards [1] Is there any way to fix these rctimes so they show the latest ctime of the subtree? Also -- are we still relying on the client clock to set the rctime / ctime of a file? Would it make sense to limit ctime/rctime for any update to the current time on the MDS ? Best Regards, Dan [1] https://github.com/ceph/ceph/pull/24023/commits/920ef964311a61fcc6c0d6671b7…

2 3

Advice needed: stuck cluster halfway upgraded, comms issues and MON space usage
by Sam Skipsey 23 Mar '21

23 Mar '21

Hi everyone: I posted to the list on Friday morning (UK time), but apparently my email is still in moderation (I have an email from the list bot telling me that it's held for moderation but no updates). Since this is a bit urgent - we have ~3PB of storage offline - I'm posting again. To save retyping the whole thing, I will direct you to a copy of the email I wrote on Friday: http://aoanla.pythonanywhere.com/Logs/EmailToCephUsers.txt (Since that was sent, we did successfully add big SSDs to the MON hosts so they don't fill up their disks with store.db s). I would appreciate any advice - assuming this also doesn't get stuck in moderation queues. -- Sam Skipsey (he/him, they/them)

3 18

Nautilus block-db resize - ceph-bluestore-tool
by Dave Hall 23 Mar '21

23 Mar '21

Hello, Based on other discussions in this list I have concluded that I need to add NVMe to my OSD nodes and expand the NVMe (DB/WAL) for each OSD. Is there a way to do this without destroying and rebuilding each OSD (after safe removal from the cluster, of course)? Is there a way to use ceph-bluestore-tool for this? Is it as simple as lvextend? Why more NVMe? Frequent DB spillovers, and the recommendation that the NVMe should be 40GB for every TB of HDD. When I did my initial setup I thought that 124GB of NVMe for a 12TB HDD would be sufficient, but by the above metric it should be more like 480GB of NVMe. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

2 1

Re: How to reset and configure replication on multiple RGW servers from scratch?
by Scheurer François 23 Mar '21

23 Mar '21

Dear All We have the same question here, if anyone can help ... Thank you! We did not find any documentation about the steps to reset & restart the sync. Especially the implications of 'bilog trim', 'mdlog trim' and 'datalog trim'. Our secondary zone is read-only. Both master and secondary zone on Nautilus (master 14.2.9 and secondary 14.2.12). Can someone also clarify following points? Many thanks in advance! 1) Is it safe to use these 3 commands (bilog trim, mdlog trim, datalog trim) on the master ? Are the bi logs exclusively used for the sync or are they needed even without multi-site? (mdlog/datalog are obviously only for multi-site) 2) Can we run these 3 commands during the sync or do we need first to stop all instances on the secondary zone ? In the latter case, do we need to stop the client traffic and wait on md/data sync to catch up prior to stop the secondary zone instances? 3) Can we then restart the instances on the secondary zone and expect rgw sync to run correctly ? Or do we need first to run 'metadata sync init' and 'data sync init' on the secondary zone ? (to trigger a full sync) Or is it necessary to delete all rgw pools on the secondary zone ? 4) And regarding the full sync, is it verifying the full object data, or only object size and mtime? If we update the secondary zone to Nautilus 14.2.18 and enable rgw_sync_obj_etag_verify, does a full sync will also detect ETag mismatches on objects that are already present on the secondary zone? Cheers Francois ________________________________ From: ceph-users on behalf of Osiński Piotr <Piotr.Osinski(a)grupawp.pl> Sent: Saturday, June 22, 2019 11:44 AM To: ceph-users(a)lists.ceph.com Subject: [ceph-users] How to reset and configure replication on multiple RGW servers from scratch? Hi, For testing purposes, I configured RGW multisite synchronization between two ceph mimic 13.2.6 clusters (I also tried: 13.2.5). Now I want to reset all current settings and configure replication from scratch. Data(pools, buckets) on the master zone will not be deleted. What has been done: 1) Deleted the secondary zone # radosgw-admin zone delete --rgw-zone=dc2_zone 2) Removed the secondary zone from zonegroup # radosgw-admin zonegroup remove --rgw-zonegroup=master_zonegroup --rgw-zone=dc2_zone 3) Commited changes # radosgw-admin period update --commit 4) Trimmed all datalogs on master zone # radosgw-admin datalog trim --start-date="2019-06-12 12:01:54" --end-date="2019-06-22 12:01:56" 5) Trimmed all error sync on master zone # radosgw-admin sync error trim --start-date="2019-06-07 07:19:26" --end-date="2019-06-22 15:59:00" 6) Deleted and recreated empty pools on secondary cluster: dc2_zone.rgw.control dc2_zone.rgw.meta dc2_zone.rgw.log dc2_zone.rgw.buckets.index dc2_zone.rgw.buckets.data Should I clear any other data / metadata in the master zone? Can data be kept somewhere in the master zone that may affect the new replication statement? I'm trying to track down a problem with blocked shards synchronization. Thank you in advance for your help. Best regards, Piotr Osiński << ATT00001.txt (0.4KB) (0.4KB) >>

1 0

New Issue - Mapping Block Devices
by duluxoz 23 Mar '21

23 Mar '21

Hi All, I've got a new issue (hopefully this one will be the last). I have a working Ceph (Octopus) cluster with a replicated pool (my-pool), an erasure-coded pool (my-pool-data), and an image (my-image) created - all *seems* to be working correctly. I also have the correct Keyring specified (ceph.client.my-id.keyring). ceph -s is reporting all healthy. The ec profile (my-ec-profile) was created with: ceph osd erasure-code-profile set my-ec-profile k=4 m=2 crush-failure-domain=host The replicated pool was created with: ceph osd pool create my-pool 100 100 replicated Followed by: rbd pool init my-pool The ec pool was created with: ceph osd pool create my-pool-data 100 100 erasure my-ec-profile --autoscale-mode=on Followed by: rbd pool init my-pool-data The image was created with: rbd create -s 1T --data-pool my-pool-data my-pool/my-image The Keyring was created with: ceph auth get-or-create client.my-id mon 'profile rbd' osd 'profile rbd pool=my-pool' mgr 'profile rbd pool=my-pool' -o /etc/ceph/ceph.client.my-id.keyring On a centos8 client machine I have installed ceph-common, placed the Keyring file into /etc/ceph/, and run the command: rbd device map my-pool/my-image --id my-id All *seems* AOK. However - and here's my issue - when I try to create a partition on /dev/rbd0 and/or try to mount it, the client reports: fdisk: cannot open /dev/rbd0: Input/output error OR mount: /my-rbd-bloc-device: special device /dev/rbd0 does not exist (respectively). What am I doing wrong? Thanks in advance for the help Matthew J -- Peregrine IT Signature *Matthew J BLACK* M.Inf.Tech.(Data Comms) MBA B.Sc. MACS (Snr), CP, IP3P When you want it done /right/ ‒ the first time! Phone: +61 4 0411 0089 Email: matthew(a)peregrineit.net <mailto:matthew@peregrineit.net> Web: www.peregrineit.net <http://www.peregrineit.net> View Matthew J BLACK's profile on LinkedIn <http://au.linkedin.com/in/mjblack> This Email is intended only for the addressee. Its use is limited to that intended by the author at the time and it is not to be distributed without the author’s consent. You must not use or disclose the contents of this Email, or add the sender’s Email address to any database, list or mailing list unless you are expressly authorised to do so. Unless otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the contents of this Email except where subsequently confirmed in writing. The opinions expressed in this Email are those of the author and do not necessarily represent the views of Peregrine I.T. Pty Ltd. This Email is confidential and may be subject to a claim of legal privilege. If you have received this Email in error, please notify the author and delete this message immediately.

4 4

How to know which client hold the lock of a file
by Norman.Kern 23 Mar '21

23 Mar '21

Hi, Anyone knows how to know which client hold lock of a file in Ceph fs? I met a dead lock problem that a client holding on get the lock, but I don't kown which client held it.

2 1