- ceph-users - lists.ceph.io

libRADOS semantics
by Cary FitzHugh 26 Jan '21

26 Jan '21

are there any good documents on the implicit requirements for librados calls? I can write a file, if it doesn't exist successfully. I can write_full a file, and it works (but many of my files will need to be chunked off disk, not enough RAM) I can append to a nonexistent file and it works. I *cannot* write to a part of an existing file (overwrite a section of it or write past it's current size) --> Errno 95 I *cannot* append to an existing file --> Errno 95 I *cannot* trunc an existing file (nor a non existent file) --> Errno 95 (what is the point of this then??) any ideas / where can one see more detailed semantics of what operations are allowed and when? The librados C API does not really go into any detail about errors and why things happen as they do. This behavior seems similar whether I am admin or a separate user. Thanks for any pointers. Cary

1 1

Auth Questions w/ librados
by Cary FitzHugh 26 Jan '21

26 Jan '21

Hello! I've got a new cluster set up. I created a new user and pool. Creating the user like so: ceph auth add client.myclient mon 'allow r' osd 'allow rw pool=testpool' I have set up librados and python bindings, with a ceph.conf and a keyring file. My trouble is that when I connect with client.admin, things seem to work correctly. but my client.myuser cannot connect.. When I try to connect with my client.myclient I get a RADOS permission denied. cluster = rados.Rados(conffile=conffile.name) My keyring only has one user in it. The conffile only has a [global] section with a mon_host string and the keyring path. I am also unable to connect to the cluster using that user with the ceph CLI client. I have had limited success in debugging any of these situations - all the debugging I might put into the ceph.conf file seems to only work after the client is connected. In testing, I also made the client.myclient user have allow * on all capabilities, and that did nothing that I could see. If anyone has thoughts on where I am stumbling, or how to debug things, that would be much appreciated. It feels like it's a combination of a few small mistakes, but I can't get a handle on the issues. Thank you. Cary

1 1

rados bench error after running vstart script- HELP PLEASE
by Bobby 26 Jan '21

26 Jan '21

Hello, I am having an error while trying to run rados benchmark after running vstart script I run : ../src/vstart.sh -d -n -l and then when I try to run: bin/rados -p rbd bench 30 write it gives me error saying: error opening pool rbd: (2) No such file or directory Can someone please help me in this. I will be grateful Thanks

3 4

Running ceph cluster on different os
by Szabo, Istvan (Agoda) 26 Jan '21

26 Jan '21

Hi, Is there anybody running a cluster with different os? Due to the centos 8 change I might try to add ubuntu osd nodes to centos cluster and decommission the centos slowly but I'm not sure this is possible or not. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

4 3

Permissions for OSD
by George Shuklin 25 Jan '21

25 Jan '21

Docs for permissions are super vague. What each flag does? What is 'x' permitting? What's the difference between class-write and write? And the last question: can we limit user to reading/writing only to existing objects in the pool? Thanks!

1 0

January Ceph Science Virtual User Group Meeting
by Kevin Hrpcek 25 Jan '21

25 Jan '21

Hey all, We will be having a Ceph science/research/big cluster call on Wednesday January 27th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. https://pad.ceph.com/p/Ceph_Science_User_Group_20210127 <https://pad.ceph.com/p/Ceph_Science_User_Group_20200923> We try to keep it to an hour or less. Ceph calendar event details: January 27th, 2020 15:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS Space Science & Engineering Center University of Wisconsin-Madison

1 0

Multisite bucket data inconsistency
by Szabo, Istvan (Agoda) 25 Jan '21

25 Jan '21

Hi, We have bucket sync enabled and seems like it is inconsistent ☹ This is the master zone sync status on that specific bucket: realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm) zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data) zone 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg) metadata sync no sync (zone is master) data sync source: 61c9d940-fde4-4bed-9389-edc8d7741817 (sin) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 126 shards behind shards: [0,1,2,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-25T11:32:57.726042+0700 [62] 104 shards are recovering recovering shards: [0,2,3,4,5,7,8,9,10,11,12,13,15,16,17,18,19,20,21,22,24,25,26,27,28,29,31,32,33,36,37,38,39,40,42,43,44,45,47,50,51,52,53,54,55,57,58,61,63,65,66,67,68,69,70,71,72,73,74,75,76,78,80,81,82,83,84,85,87,88,90,92,93,95,96,97,98,99,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,123,124,125,126,127] This is the secondary zone where the data has been uploaded: realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm) zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data) zone f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 61c9d940-fde4-4bed-9389-edc8d7741817 (sin) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source source: 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 125 shards behind shards: [0,1,2,3,4,5,6,8,9,10,11,12,13,14,15,16,17,18,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-25T11:29:32.450031+0700 [61] 126 shards are recovering recovering shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,104,105,106,107,108,109,110,111,112,114,115,116,117,118,119,120,121,122,123,124,125,126,127] The pipes are already there: "id": "seo-2", "data_flow": { "symmetrical": [ { "id": "seo-2-flow", "zones": [ "9213182a-14ba-48ad-bde9-289a1c0c0de8", "f20ddd64-924b-4f78-8d2d-dd6c65f98ba9" ] } ] }, "pipes": [ { "id": "seo-2-hkg-ash-pipe", "source": { "bucket": "seo..prerender", "zones": [ "9213182a-14ba-48ad-bde9-289a1c0c0de8" ] }, "dest": { "bucket": "seo..prerender", "zones": [ "f20ddd64-924b-4f78-8d2d-dd6c65f98ba9" ] }, "params": { "source": { "filter": { "tags": [] } }, "dest": {}, "priority": 0, "mode": "system", "user": "" } }, { "id": "seo-2-ash-hkg-pipe", "source": { "bucket": "seo..prerender", "zones": [ "f20ddd64-924b-4f78-8d2d-dd6c65f98ba9" ] }, "dest": { "bucket": "seo..prerender", "zones": [ "9213182a-14ba-48ad-bde9-289a1c0c0de8" ] }, "params": { "source": { "filter": { "tags": [] } }, "dest": {}, "priority": 0, "mode": "system", "user": "" } } ], "status": "enabled" } Any idea to troubleshoot? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 2

cephadm db_slots and wal_slots ignored
by Schweiss, Chip 25 Jan '21

25 Jan '21

I'm trying to set up a new ceph cluster with cephadm on a SUSE SES trial that has Ceph 15.2.8 Each OSD node has 18 rotational SAS disks, 4 NVMe 2TB SSDs for DB, and 2 NVME2 200GB Optane SSDs for WAL. These servers will eventually have 24 rotational SAS disks that they will inherit from existing storage servers. So I don't want all the space used on the DB and WAL SSDs. I suspect from the comment "(db_slots is actually to be favoured here, but it's not implemented yet)" on this page in the docs: https://docs.ceph.com/en/latest/cephadm/drivegroups/#the-advanced-case these parameters are not yet implemented, yet are documented as such under "ADDITIONAL OPTIONS" My osd_spec.yml: service_type: osd service_id: three_tier_osd placement: host_pattern: '*' data_devices: rotational: 1 model: 'ST14000NM0288' db_devices: rotational: 0 model: 'INTEL SSDPE2KX020T8' limit: 6 wal_devices: model: 'INTEL SSDPEL1K200GA' limit: 12 db_slots: 6 wal_slots: 12 All available space is consumed on my DB and WAL SSDs with only 18 OSDs, leaving no room to add additional spindles. Is this still work in progress, or a bug I should report? Possibly related to https://github.com/rook/rook/issues/5026 At the minimum, this appears to be a documentation bug. How can I work around this? -Chip

2 3

radosgw-admin realm pull from the secondary site fails "(13) Permission denied"
by Hayashida, Mami 23 Jan '21

23 Jan '21

I have been trying to create two virtual test clusters to learn about the RGW multisite setting. So far, I have set up two small Nautilus (v.14.2.16) clusters, designated one of them as the "master zone site" and followed every step outlined in the doc ( https://docs.ceph.com/en/nautilus/radosgw/multisite/) including create a system user, updating the period, and restarting the rgw daemon. (For the sake of simplicity, there is only one RGW daemon running on each site.) Once I installed the RGW daemon on the secondary zone site, I tried pulling the realm from the master zone cluster, but ended up with this: ``` $ radosgw-admin realm pull --url=http://<master zone gateway>:80 --access-key=<system_access_key> --secret=<system_secret_key> request failed: (13) Permission denied If the realm has been changed on the master zone, the master zone's gateway may need to be restarted to recognize this user. ``` I tried adding the --rgw-realm=<realm set up in the primary site>, but the result was the same. I restarted the rgw daemon on both sides -- that did not help, either. The output of all of the following on the master zone side, as far as I could tell, seems correct -- the realm, zonegroup, zone I created are the only ones and set to default. ``` radosgw-admin zone/zonegroup/realm list radosgw-admin zone/zonegroup/realm get ``` On the "master zone" side, the rgw log shows ``` 2021-01-22 13:34:48.404 7fb9ca89e700 1 ====== starting new request req=0x7fb9ca897740 ===== 2021-01-22 13:34:48.428 7fb9ca89e700 1 ====== req done req=0x7fb9ca897740 op status=0 http_status=403 latency=0.0240002s ====== 2021-01-22 13:34:48.428 7fb9ca89e700 1 civetweb: 0x559d6509a000: 10.33.30.55 - - [22/Jan/2021:13:34:48 -0500] "GET /admin/realm HTTP/1.1" 403 318 - - ``` I am using Ubuntu 18.04, Ceph v.14.2.16, deployed using `ceph-deploy`. *Mami Hayashida* *Research Computing Associate* Univ. of Kentucky ITS Research Computing Infrastructure

2 2

Re: Storage down due to MON sync very slow
by Frank Schilder 22 Jan '21

22 Jan '21

Hi Dan, it is possible that the payload reduction also solved or at least reduced a really bad problem that looks related (beware, that's a long one): https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/FBGIJZNFG44… . Since reducing the payload size I still observe these large peaks in the MON network activity. However, it seems that the cluster does not go down like before any more. During these peaks, I see warnings like these: 2021-01-22 12:00:00.000102 [WRN] overall HEALTH_WARN 1 pools nearfull 2021-01-22 11:04:09.156796 [INF] Health check cleared: SLOW_OPS (was: 5 slow ops, oldest one blocked for 75 sec, mon.ceph-02 has slow ops) 2021-01-22 11:04:07.994416 [WRN] Health check update: 5 slow ops, oldest one blocked for 75 sec, mon.ceph-02 has slow ops (SLOW_OPS) 2021-01-22 11:04:01.469498 [WRN] Health check failed: 124 slow ops, oldest one blocked for 82 sec, daemons [mon.ceph-02,mon.ceph-03] have slow ops. (SLOW_OPS) 2021-01-22 11:00:00.000104 [WRN] overall HEALTH_WARN 1 pools nearfull 2021-01-22 10:36:44.576663 [INF] Health check cleared: SLOW_OPS (was: 25 slow ops, oldest one blocked for 42 sec, daemons [mon.ceph-02,mon.ceph-03] have slow ops.) 2021-01-22 10:36:38.543763 [WRN] Health check failed: 18 slow ops, oldest one blocked for 38 sec, daemons [mon.ceph-02,mon.ceph-03] have slow ops. (SLOW_OPS) So, at least stuff is working. I now lean towards the hypothesis that these outages were caused by some synchronisation process between MONs that got less problematic with reducing the payload size. I might be able to reduce my insane beacon time-outs again, but before doing so, do you know of any other communication parameters similar to the mon_sync_max_payload_size that might be relevant in MON-[MON, MGR, OSD] communication? In general, I have the impression that due to such little bugs the recommendation for production clusters should be elevated to at least 5 MONs so that one can afford 2 MONs going out of quorum temporarily. I will upgrade our cluster to 5 MONs as soon as I can. Thanks for your help and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dan(a)vanderster.com> Sent: 06 January 2021 20:53:14 To: Frank Schilder Subject: Re: [ceph-users] Re: Storage down due to MON sync very slow Yeah I was going to say -- ignore all of the rsync advice in that thread, it is unnecessary. Setting a small mon sync payload works like magic :) -- dan On Wed, Jan 6, 2021 at 8:49 PM Frank Schilder <frans(a)dtu.dk> wrote: > > OK, sorry for all my questions. > > Setting mon_sync_max_payload_size=4096 actually makes the MON sync in no time! Thank you so much :) > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Frank Schilder > Sent: 06 January 2021 20:40:26 > To: Dan van der Ster > Subject: Re: [ceph-users] Re: Storage down due to MON sync very slow > > OK, thanks a lot! I will try it now. Hope the cluster remains responsive. > > I'm wondering about this approach someone brought up in your thread: > > Eventually I stopped one MON, tarballed it's database and used that to > bring back the MON which was upgraded to 13.2.8 > > That work without any hickups. The MON joined again within a few seconds. > > Stopping one MON for a copy would be much shorter storage outage than the sync I'm doing. I guess its the entire mon data directory to copy. I always wondered if this contains data tied to a specific MON. If not, the copy approach could speed things up a lot. What do you think? > > Thanks again and best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Dan van der Ster <dan(a)vanderster.com> > Sent: 06 January 2021 20:36:15 > To: Frank Schilder > Subject: Re: [ceph-users] Re: Storage down due to MON sync very slow > > We have used mon_sync_max_payload_size 4096 on our largest most > important prod cluster since that thread. > The PR from Sage makes something like that the default anyway. (the PR > counts keys rather than bytes, but the effect is the same). > > mon_sync_max_payload_size 4096 should not impact the speed of syncing > -- it simply breaks the sync into smaller more manageable pieces. > (Without this, if you have lots of keys in the mon db, in our case > caused by lots of rbd snapshots, then syncing will never ever > complete). > > -- dan > > On Wed, Jan 6, 2021 at 8:32 PM Frank Schilder <frans(a)dtu.dk> wrote: > > > > Hi Dan, > > > > thanks for that. Will it slow down or accelerate the syncing (will read your post after that e-mail), or will it just allow I/O to continue and sync more in the background? Current value is > > > > mon_sync_max_payload_size 1048576 > > > > Related to that, would building a MON store from OSDs following https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#… help providing a head start? Not sure if this procedure works on an active cluster. > > > > Will study your thread now ... > > > > Thanks again and best regards, > > ================= > > Frank Schilder > > AIT Risø Campus > > Bygning 109, rum S14 > > > > ________________________________________ > > From: Dan van der Ster <dan(a)vanderster.com> > > Sent: 06 January 2021 20:26:46 > > To: Frank Schilder > > Subject: Re: [ceph-users] Re: Storage down due to MON sync very slow > > > > (obviously just put that config in the ceph.conf on the mons if mimic > > doesn't have ceph config... I don't quite remember.) > > > > -- dan > > > > On Wed, Jan 6, 2021 at 8:25 PM Dan van der Ster <dan(a)vanderster.com> wrote: > > > > > > This sounds a lot like an old thread of mine: > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/M5ZKF7PTEO2… > > > > > > See the discussion about mon_sync_max_payload_size, and the PR that > > > fixed this at some point in nautilus. > > > > > > Our workaround was: > > > > > > ceph config set mon mon_sync_max_payload_size 4096 > > > > > > Hope that helps, > > > > > > Dan > > > > > > > > > On Wed, Jan 6, 2021 at 8:18 PM Frank Schilder <frans(a)dtu.dk> wrote: > > > > > > > > Dear Dan, > > > > > > > > thanks for your fast response. > > > > > > > > Version: mimic 13.2.10. > > > > > > > > Here is the mon_status of the "new" MON during syncing: > > > > > > > > [root@ceph-01 ~]# ceph daemon mon.ceph-01 mon_status > > > > { > > > > "name": "ceph-01", > > > > "rank": 0, > > > > "state": "synchronizing", > > > > "election_epoch": 0, > > > > "quorum": [], > > > > "features": { > > > > "required_con": "144115188346404864", > > > > "required_mon": [ > > > > "kraken", > > > > "luminous", > > > > "mimic", > > > > "osdmap-prune" > > > > ], > > > > "quorum_con": "0", > > > > "quorum_mon": [] > > > > }, > > > > "outside_quorum": [ > > > > "ceph-01" > > > > ], > > > > "extra_probe_peers": [], > > > > "sync_provider": [], > > > > "sync": { > > > > "sync_provider": "mon.2 192.168.32.67:6789/0", > > > > "sync_cookie": 33302773774, > > > > "sync_start_version": 38355711 > > > > }, > > > > "monmap": { > > > > "epoch": 3, > > > > "fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9", > > > > "modified": "2019-03-14 23:08:34.717223", > > > > "created": "2019-03-14 22:18:15.088212", > > > > "features": { > > > > "persistent": [ > > > > "kraken", > > > > "luminous", > > > > "mimic", > > > > "osdmap-prune" > > > > ], > > > > "optional": [] > > > > }, > > > > "mons": [ > > > > { > > > > "rank": 0, > > > > "name": "ceph-01", > > > > "addr": "192.168.32.65:6789/0", > > > > "public_addr": "192.168.32.65:6789/0" > > > > }, > > > > { > > > > "rank": 1, > > > > "name": "ceph-02", > > > > "addr": "192.168.32.66:6789/0", > > > > "public_addr": "192.168.32.66:6789/0" > > > > }, > > > > { > > > > "rank": 2, > > > > "name": "ceph-03", > > > > "addr": "192.168.32.67:6789/0", > > > > "public_addr": "192.168.32.67:6789/0" > > > > } > > > > ] > > > > }, > > > > "feature_map": { > > > > "mon": [ > > > > { > > > > "features": "0x3ffddff8ffacfffb", > > > > "release": "luminous", > > > > "num": 1 > > > > } > > > > ], > > > > "mds": [ > > > > { > > > > "features": "0x3ffddff8ffacfffb", > > > > "release": "luminous", > > > > "num": 2 > > > > } > > > > ], > > > > "client": [ > > > > { > > > > "features": "0x2f018fb86aa42ada", > > > > "release": "luminous", > > > > "num": 1 > > > > }, > > > > { > > > > "features": "0x3ffddff8eeacfffb", > > > > "release": "luminous", > > > > "num": 1 > > > > }, > > > > { > > > > "features": "0x3ffddff8ffacfffb", > > > > "release": "luminous", > > > > "num": 17 > > > > } > > > > ] > > > > } > > > > } > > > > > > > > I'm a bit surprised that the other 2 MONs don't remain in quorum until this MON has caught up. Is there any way to monitor the syncing progress? Right now I need to interrupt regularly to allow some I/O, but I have no clue how long I need to wait. > > > > > > > > Thanks for your help! > > > > > > > > Best regards, > > > > ================= > > > > Frank Schilder > > > > AIT Risø Campus > > > > Bygning 109, rum S14 > > > > > > > > ________________________________________ > > > > From: Dan van der Ster <dan(a)vanderster.com> > > > > Sent: 06 January 2021 20:16:44 > > > > To: Frank Schilder > > > > Cc: Ceph Users > > > > Subject: Re: [ceph-users] Re: Storage down due to MON sync very slow > > > > > > > > Which version of Ceph are you running? > > > > > > > > .. dan > > > > > > > > > > > > On Wed, Jan 6, 2021, 8:14 PM Frank Schilder <frans(a)dtu.dk<mailto:frans@dtu.dk>> wrote: > > > > In the output of the MON I see slow ops warnings: > > > > > > > > debug 2021-01-06 20:12:48.854 7f1a3d29f700 -1 mon.ceph-01@0(synchronizing) e3 get_health_metrics reporting 20 slow ops, oldest is log(1 entries from seq 1 at 2021-01-06 20:00:12.014861) > > > > > > > > There appears to be no progress on this operation, it is stuck. > > > > > > > > Best regards, > > > > ================= > > > > Frank Schilder > > > > AIT Risø Campus > > > > Bygning 109, rum S14 > > > > > > > > ________________________________________ > > > > From: Frank Schilder <frans(a)dtu.dk<mailto:frans@dtu.dk>> > > > > Sent: 06 January 2021 20:11:25 > > > > To: ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> > > > > Subject: [ceph-users] Storage down due to MON sync very slow > > > > > > > > Dear all, > > > > > > > > I had to restart one out of 3 MONs on an empty MON DB dir. It is in state syncing right now, but I'm not sure if there is any progress. The cluster is completely unresponsive even though I have 2 healthy MONs. Is there any way to sync the DB directory faster and/or without downtime? > > > > > > > > Thanks a lot! > > > > > > > > Best regards, > > > > ================= > > > > Frank Schilder > > > > AIT Risø Campus > > > > Bygning 109, rum S14 > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io> > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 0