- ceph-users - lists.ceph.io

Ceph User Survey Now Available
by Mike Perez 09 Mar '21

09 Mar '21

Hi everyone! Be sure to make your voice heard by taking the Ceph User Survey before April 2, 2021. This information will help guide the Ceph community’s investment in Ceph and the Ceph community's future development. https://ceph.io/user-survey/ Thank you to the Ceph User Survey Working Group for designing this year's survey! * Anantha Adiga * Anthony D'Atri * Paul Mezzanini * Stefan Kooman https://tracker.ceph.com/projects/ceph/wiki/User_Survey_Working_Group We will provide the final results in the coming months after the survey has ended. -- Mike Perez

1 1

PG stuck at active+clean+remapped
by Michael Fladischer 09 Mar '21

09 Mar '21

Hi, we have replaced some of our OSDs a while ago an while everything recovery as planned, one PG is still stuck at active+clean+remapped with no backfilling taking place. Mpaaing the PG in question shows me that one OSD is missing: $ ceph pg map 35.1fe osdmap e1265760 pg 35.1fe (35.1fe) -> up [97,190,65,23,393,223,2147483647,354,132] acting [97,190,65,23,393,223,112,354,132] It seems that osd.112 should be replaced with an other OSD and I suspect that CRUSH cannot find a suitable one. Pool 35 is EC and has k=7 and m=2 and our Cluster has 9 OSD nodes. Is this just a case of CRUSH giving up to early as described in the troubleshooting PGs section[0] of the docs? Running the test as described there using `crushtool` gives several bad mapping rule results for "--num-rep 9". If so, would it help to just add new OSDs to the existing hosts or would it be better to add a whole new OSD host? Are there other options (upmap) to force this single PG to use a different set of OSDs for its "up" map? [0] https://github.com/ceph/ceph/blob/master/doc/rados/troubleshooting/troubles… Thanks, Michael

1 0

node down pg with backfill_wait waiting for incomplete?
by Marc 09 Mar '21

09 Mar '21

I have a node down and pg's are remapping/backfilling. I have also a lot of pg's in backfill_wait. I was wondering if there is a specific order that this is being executed. Eg I have a large'garbage' pool ec21 that is stuck. I could resolve that by changing the min size. However I rather have this not remapped, and wait for this node to be back on line. Question is: is other remapping/backfilling waiting for this stuck to be fixed, or is backfilling/remapping so smart that it will do what ever it can? - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -. F1 Outsourcing Development Sp. z o.o. Poland t: +48 (0)12 4466 845 f: +48 (0)12 4466 843 e: marc(a)f1-outsourcing.eu

1 0

Rados gateway basic pools missing
by St-Germain, Sylvain (SSC/SPC) 09 Mar '21

09 Mar '21

Hi everyone, I just rebuild a (test) cluster using : OS : Ubuntu 20.04.2 LTS CEPH : ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable) 3 nodes : monitor/storage 1. The cluster looks good : # ceph -s cluster: id: 9a89aa5a-1702-4f87-a99c-f94c9f2cdabd health: HEALTH_OK services: mon: 3 daemons, quorum dao-wkr-04,dao-wkr-05,dao-wkr-06 (age 7m) mgr: dao-wkr-05(active, since 8m), standbys: dao-wkr-04, dao-wkr-06 mds: cephfs:1 {0=dao-wkr-04=up:active} 2 up:standby osd: 9 osds: 9 up (since 7m), 9 in (since 4h) rgw: 3 daemons active (dao-wkr-04.rgw0, dao-wkr-05.rgw0, dao-wkr-06.rgw0) task status: data: pools: 7 pools, 121 pgs objects: 234 objects, 16 KiB usage: 9.0 GiB used, 2.0 TiB / 2.0 TiB avail pgs: 121 active+clean 2. except that the main pools for the radosgw are not there # sudo ceph osd lspools 1 device_health_metrics 2 cephfs_data 3 cephfs_metadata 4 .rgw.root 5 default.rgw.log 6 default.rgw.control 7 default.rgw.meta Missing : default.rgw.buckets.index & default.rgw.buckets.data What do you think ? Thx ! Sylvain

1 1

Replacing disk with xfs on it, documentation?
by Drew Weaver 09 Mar '21

09 Mar '21

Hello, I haven't needed to replace a disk in awhile and it seems that I have misplaced my quick little guide on how to do it. When searching the docs it is now recommending that you should use ceph-volume to create OSDs when doing that it creates LV: Disk /dev/sde: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mapper/ceph--34b5a0a9--f84a--416f--8b74--fb1e05161f80-osd--block--4581caf4--eef0--42e1--b237--c114dfde3d15: 4000.2 GB, 4000220971008 bytes, 7812931584 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Cool but all of my other OSDs look like this and appear to just be XFS. Disk /dev/sdd: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: gpt Disk identifier: D0195044-10B5-4113-8210-A5CFCB9213A2 # Start End Size Type Name 1 2048 206847 100M Ceph OSD ceph data 2 206848 7812939742 3.7T unknown ceph block 35075 ? S< 0:00 [xfs-buf/sdd1] 35076 ? S< 0:00 [xfs-data/sdd1] 35077 ? S< 0:00 [xfs-conv/sdd1] 35078 ? S< 0:00 [xfs-cil/sdd1] 35079 ? S< 0:00 [xfs-reclaim/sdd] 35080 ? S< 0:00 [xfs-log/sdd1] 35082 ? S 0:00 [xfsaild/sdd1] I can't seem to find the instructions in order to create an OSD like the original ones anymore. I'm pretty sure that when this cluster was setup it was setup using the: ceph-deploy osd prepare ceph-deploy osd create ceph-deploy osd activate commands, but it seems as though the prepare and activate commands have since been removed from ceph-deploy, so I am really confused. =) Does anyone by chance have the instructions for replacing a failed drive when it meets the above criteria?

1 0

DocuBetter Meeting This Week -- 10 Mar 2021 1730 UTC
by John Zachary Dover 09 Mar '21

09 Mar '21

This week's meeting will focus on the ongoing rewrite of the cephadm documentation and making certain that the documentation addresses the rough edges in the Pacific release. Meeting: https://bluejeans.com/908675367 Etherpad: https://pad.ceph.com/p/Ceph_Documentation

1 0

Bluestore OSD crash with tcmalloc::allocate_full_cpp_throw_oom in multisite setup with PG_DAMAGED cluster error
by Szabo, Istvan (Agoda) 09 Mar '21

09 Mar '21

Hi, I have a 3 DC multisite setup. The replication is directional like HKG->SGP->US so the bucket is replicated from HKG to SGP and the same bucket is replicated further from SGP to US. The HKG > SGP connection is pretty fast 12.5millions objects (600GB) transferred in 6.5 hours. Once the OSD crashed in SGP, it stopped the complete chain replication and made PG_DAMAGED cluster error. The pg can be repaired but the sync never started back only bucket sync disable/enable helped. I got OSD crash also in HKG BUT in ASH. ASH no any error, the replication speed is 2 millions objects in 6.5 hours which is like 90GB of data. This is the crash of the osd: { "backtrace": [ "(()+0x12b20) [0x7f597d3fbb20]", "(gsignal()+0x10f) [0x7f597c0667ff]", "(abort()+0x127) [0x7f597c050c35]", "(()+0x9009b) [0x7f597ca1c09b]", "(()+0x9653c) [0x7f597ca2253c]", "(()+0x96597) [0x7f597ca22597]", "(()+0x967f8) [0x7f597ca227f8]", "(()+0x19d24) [0x7f597e168d24]", "(tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x146) [0x7f597e18b0d6]", "(rocksdb::Arena::AllocateNewBlock(unsigned long)+0x43) [0x5632f08ccb93]", "(rocksdb::Arena::AllocateFallback(unsigned long, bool)+0x4b) [0x5632f08ccc3b]", "(rocksdb::ConcurrentArena::AllocateAligned(unsigned long, unsigned long, rocksdb::Logger*)+0xb4) [0x5632f07fae94]", "(()+0x1103e7e) [0x5632f085ae7e]", "(rocksdb::MemTable::Add(unsigned long, rocksdb::ValueType, rocksdb::Slice const&, rocksdb::Slice const&, bool, rocksdb::MemTablePostProcessInfo*)+0xcf) [0x5632f07f6f8f]", "(rocksdb::MemTableInserter::PutCFImpl(unsigned int, rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::ValueType)+0x452) [0x5632f08520e2]", "(rocksdb::MemTableInserter::PutCF(unsigned int, rocksdb::Slice const&, rocksdb::Slice const&)+0x17) [0x5632f0852e97]", "(rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler*) const+0x480) [0x5632f084ac20]", "(rocksdb::WriteBatchInternal::InsertInto(rocksdb::WriteThread::WriteGroup&, unsigned long, rocksdb::ColumnFamilyMemTables*, rocksdb::FlushScheduler*, bool, unsigned long, rocksdb::DB*, bool, bool, bool)+0x149) [0x5632f084ebe9]", "(rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1acd) [0x5632f078a03d]", "(rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x21) [0x5632f078ac11]", "(RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x8c) [0x5632f074180c]", "(RocksDBStore::submit_transaction(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x87) [0x5632f0742027]", "(BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)+0x426) [0x5632f0226376]", "(BlueStore::_kv_sync_thread()+0x176f) [0x5632f024bc1f]", "(BlueStore::KVSyncThread::entry()+0x11) [0x5632f0273791]", "(()+0x814a) [0x7f597d3f114a]", "(clone()+0x43) [0x7f597c12bf23]" ], "ceph_version": "15.2.9", "crash_id": "2021-03-04T14:55:45.094048Z_3d481fd3-7573-4cb7-9b22-20784b418e64", "entity_name": "osd.5", "os_id": "centos", "os_name": "CentOS Linux", "os_version": "8", "os_version_id": "8", "process_name": "ceph-osd", "stack_sig": "9643c370a20c0d34f5e8965ae4461e2a7cf709ab4183929239bc263d0e1eef94", "timestamp": "2021-03-04T14:55:45.094048Z", "utsname_hostname": "hostname", "utsname_machine": "x86_64", "utsname_release": "4.18.0-240.10.1.el8_3.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP Mon Jan 18 17:05:51 UTC 2021" } Any idea what I should tune? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

ny way can recover data?
by Elians Wan 09 Mar '21

09 Mar '21

we use s3 api upload file to ceph cluster with non versioned bucket, and we override many files then want to recover them, any way can recover?

1 0

2 Pgs (1x inconsistent, 1x unfound / degraded - unable to fix
by Jeremi Avenant 09 Mar '21

09 Mar '21

Good day I'm currently decommissioning a cluster that runs EC3+1 (rack failure domain - with 5 racks), however the cluster still has some production items on it since I'm in the process of moving it to our new EC8+2 cluster. Running Luminous 12.2.13 on Ubuntu 16 HWE, containerized with ceph-ansible 3.2. I currently get the following error after we lost 1 OSD (195). I'm forced to repair, scrub, deep scrub, restart OSDs etc, everything mentioned in the troubleshooting docs & information from IRC but cannot for the life of me get it to work. What I'm seeing is, that pg 9.3dd (volume_images) has a status of 1 OSD (osd is down) which I know, but the other OSD shows 316 (not queried). Also pg 9.3dd has 4x functioning UP's but still reference the missing OSD under acting. Regards OBJECT_UNFOUND 1/501815192 objects unfound (0.000%) pg 9.3dd has 1 unfound objects OSD_SCRUB_ERRORS 1 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 9.3d1 is active+clean+inconsistent, acting [347,316,307,249] PG_DEGRADED Degraded data redundancy: 1219/2001837265 objects degraded (0.000%), 1 pg degraded, 1 pg undersized pg 9.3dd is stuck undersized for 55486.439002, current state active+recovery_wait+forced_recovery+undersized+degraded+remapped, last acting [355,2147483647,64,367] ceph pg 9.3dd query "up": [ 355, 139, 64, 367 ], "acting": [ 355, 2147483647, 64, 367 "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2021-03-08 16:07:51.239010", "might_have_unfound": [ { "osd": "64(2)", "status": "already probed" }, { "osd": "139(1)", "status": "already probed" }, { "osd": "195(1)", "status": "osd is down" }, { "osd": "316(2)", "status": "not queried" }, { "osd": "367(3)", "status": "already probed" } ], "recovery_progress": { "backfill_targets": [ "139(1)" ceph pg 9.3d1 query { "state": "active+clean+inconsistent", "snap_trimq": "[]", "snap_trimq_len": 0, "epoch": 168488, "up": [ 347, 316, 307, 249 ], "acting": [ 347, 316, 307, 249 ], "actingbackfill": [ "249(3)", "307(2)", "316(1)", "347(0)" -- CLUSTER STATS: cluster: id: 1ea59fbe-46a4-474e-8225-a66b32ca86b7 health: HEALTH_ERR 1/490166525 objects unfound (0.000%) 1 scrub errors Possible data damage: 1 pg inconsistent Degraded data redundancy: 1259/1956055233 objects degraded (0.000%), 1 pg degraded, 1 pg undersized services: mon: 3 daemons, quorum B-04-11-cephctl,B-05-11-cephctl,B-03-11-cephctl mgr: B-03-11-cephctl(active), standbys: B-04-11-cephctl, B-05-11-cephctl mds: cephfs-1/1/1 up {0=B-04-11-cephctl=up:active}, 2 up:standby osd: 384 osds: 383 up, 383 in; 1 remapped pgs data: pools: 11 pools, 13440 pgs objects: 490.17M objects, 1.35PiB usage: 1.88PiB used, 2.33PiB / 4.21PiB avail pgs: 1259/1956055233 objects degraded (0.000%) 1/490166525 objects unfound (0.000%) 13332 active+clean 96 active+clean+scrubbing+deep 10 active+clean+scrubbing 1 active+clean+inconsistent 1 active+recovery_wait+forced_recovery+undersized+degraded+remapped *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za <http://www.uwc.ac.za/> E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

1 0

Any way can recover data?
by Elians Wan 09 Mar '21

09 Mar '21

we use s3 api upload file to ceph cluster with non versioned bucket, and we override many files then want to recover them, any way can recover?

1 0