March 2021 - ceph-users - lists.ceph.io

by Boris Behrens

Good morning ceph people, I have a pool that got a whitespace as name. And I want to know what creates the pool. I already renamed it, but something recreates the pool. Is there a way to find out what created the pool and what the content ist? When I checked it's content I get [root@s3db1 ~]# rados -p " " ls .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1 .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86469149.1766:_Feueqyht8adm09IVoqhR7Sh:1 .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86450896.1831:_rU77PL7UJkTlOovtCWkvnfP:1 ... [root@s3db1 ~]# rados -p " " listxattr .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1 ceph.objclass.version user.rgw.acl [root@s3db1 ~]# rados -p " " stat .meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1 /.meta:bucket.instance:temonitor:ff7a8b0c-07e6-463a-861b-78f0adeba8ad.86912481.1982:_vgJnkKGi8y7roIM45BENWjD:1 mtime 2021-03-10 06:27:43.000000, size 317 -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

3 years, 1 month

1
3
0 0

Many OSD marked down after no beacon for XXX seconds, just becauseone MON's OS disk was blocked.

by 912273695＠qq.com

Hello，everyone, In the OS disk blocked scene, the Mon service is still running ,but cant't work normally, soon the mon will out of quorum, but some OSD were still markd down after mon_osd_report_timeout*2 seconds, which will cause the cluster unavailable. At this time， the OS is very slow，may be unable to operate or login，so it‘s impossible to kill the mon by hand or other service. This problem exist in L and N version. Any suggestion？ you can reproduce this problem as follow: 1、 block the OS disk use follow command. # echo blocked > /sys/block/sdx/device/state 2、a few seconds later， the mon will out of quorum 3、after mon_osd_report_timeout*2 seconds，some osds will be markd down. （mon_osd_report_timeout default value 900s，you can set a smaller value） 912273695(a)qq.com

3 years, 1 month

1
0
0 0

February 2021 Tech Talk and Code Walk-through

by Mike Perez

Hi everyone! I'm excited to announce two talks we have on the schedule for February 2021: Jason Dillaman will be giving part 2 to the librbd code walk-through. The stream starts on February 23rd at 18:00 UTC / 19:00 CET / 1:00 PM EST / 10:00 AM PST https://tracker.ceph.com/projects/ceph/wiki/Code_Walkthroughs Part 1: https://www.youtube.com/watch?v=L0x61HpREy4 -------------- What's New in the Pacific Release Hear Sage Weil give a live update on the development of the Pacific Release. The stream starts on February 25th at 17:00 UTC / 18:00 CET / 12 PM EST / 9 AM PST. https://ceph.io/ceph-tech-talks/ All live streams will be recorded and -- Mike Perez

3 years, 1 month

1
2
0 0

Ceph User Survey Now Available

by Mike Perez

Hi everyone! Be sure to make your voice heard by taking the Ceph User Survey before April 2, 2021. This information will help guide the Ceph community’s investment in Ceph and the Ceph community's future development. https://ceph.io/user-survey/ Thank you to the Ceph User Survey Working Group for designing this year's survey! * Anantha Adiga * Anthony D'Atri * Paul Mezzanini * Stefan Kooman https://tracker.ceph.com/projects/ceph/wiki/User_Survey_Working_Group We will provide the final results in the coming months after the survey has ended. -- Mike Perez

3 years, 1 month

1
1
0 0

PG stuck at active+clean+remapped

by Michael Fladischer

Hi, we have replaced some of our OSDs a while ago an while everything recovery as planned, one PG is still stuck at active+clean+remapped with no backfilling taking place. Mpaaing the PG in question shows me that one OSD is missing: $ ceph pg map 35.1fe osdmap e1265760 pg 35.1fe (35.1fe) -> up [97,190,65,23,393,223,2147483647,354,132] acting [97,190,65,23,393,223,112,354,132] It seems that osd.112 should be replaced with an other OSD and I suspect that CRUSH cannot find a suitable one. Pool 35 is EC and has k=7 and m=2 and our Cluster has 9 OSD nodes. Is this just a case of CRUSH giving up to early as described in the troubleshooting PGs section[0] of the docs? Running the test as described there using `crushtool` gives several bad mapping rule results for "--num-rep 9". If so, would it help to just add new OSDs to the existing hosts or would it be better to add a whole new OSD host? Are there other options (upmap) to force this single PG to use a different set of OSDs for its "up" map? [0] https://github.com/ceph/ceph/blob/master/doc/rados/troubleshooting/troubles… Thanks, Michael

3 years, 1 month

1
0
0 0

node down pg with backfill_wait waiting for incomplete?

by Marc

I have a node down and pg's are remapping/backfilling. I have also a lot of pg's in backfill_wait. I was wondering if there is a specific order that this is being executed. Eg I have a large'garbage' pool ec21 that is stuck. I could resolve that by changing the min size. However I rather have this not remapped, and wait for this node to be back on line. Question is: is other remapping/backfilling waiting for this stuck to be fixed, or is backfilling/remapping so smart that it will do what ever it can? - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -. F1 Outsourcing Development Sp. z o.o. Poland t: +48 (0)12 4466 845 f: +48 (0)12 4466 843 e: marc(a)f1-outsourcing.eu

3 years, 1 month

1
0
0 0

Rados gateway basic pools missing

by St-Germain, Sylvain (SSC/SPC)

Hi everyone, I just rebuild a (test) cluster using : OS : Ubuntu 20.04.2 LTS CEPH : ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable) 3 nodes : monitor/storage 1. The cluster looks good : # ceph -s cluster: id: 9a89aa5a-1702-4f87-a99c-f94c9f2cdabd health: HEALTH_OK services: mon: 3 daemons, quorum dao-wkr-04,dao-wkr-05,dao-wkr-06 (age 7m) mgr: dao-wkr-05(active, since 8m), standbys: dao-wkr-04, dao-wkr-06 mds: cephfs:1 {0=dao-wkr-04=up:active} 2 up:standby osd: 9 osds: 9 up (since 7m), 9 in (since 4h) rgw: 3 daemons active (dao-wkr-04.rgw0, dao-wkr-05.rgw0, dao-wkr-06.rgw0) task status: data: pools: 7 pools, 121 pgs objects: 234 objects, 16 KiB usage: 9.0 GiB used, 2.0 TiB / 2.0 TiB avail pgs: 121 active+clean 2. except that the main pools for the radosgw are not there # sudo ceph osd lspools 1 device_health_metrics 2 cephfs_data 3 cephfs_metadata 4 .rgw.root 5 default.rgw.log 6 default.rgw.control 7 default.rgw.meta Missing : default.rgw.buckets.index & default.rgw.buckets.data What do you think ? Thx ! Sylvain

3 years, 1 month

1
1
0 0

Replacing disk with xfs on it, documentation?

by Drew Weaver

Hello, I haven't needed to replace a disk in awhile and it seems that I have misplaced my quick little guide on how to do it. When searching the docs it is now recommending that you should use ceph-volume to create OSDs when doing that it creates LV: Disk /dev/sde: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mapper/ceph--34b5a0a9--f84a--416f--8b74--fb1e05161f80-osd--block--4581caf4--eef0--42e1--b237--c114dfde3d15: 4000.2 GB, 4000220971008 bytes, 7812931584 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Cool but all of my other OSDs look like this and appear to just be XFS. Disk /dev/sdd: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: gpt Disk identifier: D0195044-10B5-4113-8210-A5CFCB9213A2 # Start End Size Type Name 1 2048 206847 100M Ceph OSD ceph data 2 206848 7812939742 3.7T unknown ceph block 35075 ? S< 0:00 [xfs-buf/sdd1] 35076 ? S< 0:00 [xfs-data/sdd1] 35077 ? S< 0:00 [xfs-conv/sdd1] 35078 ? S< 0:00 [xfs-cil/sdd1] 35079 ? S< 0:00 [xfs-reclaim/sdd] 35080 ? S< 0:00 [xfs-log/sdd1] 35082 ? S 0:00 [xfsaild/sdd1] I can't seem to find the instructions in order to create an OSD like the original ones anymore. I'm pretty sure that when this cluster was setup it was setup using the: ceph-deploy osd prepare ceph-deploy osd create ceph-deploy osd activate commands, but it seems as though the prepare and activate commands have since been removed from ceph-deploy, so I am really confused. =) Does anyone by chance have the instructions for replacing a failed drive when it meets the above criteria?

3 years, 1 month

1
0
0 0

DocuBetter Meeting This Week -- 10 Mar 2021 1730 UTC

by John Zachary Dover

This week's meeting will focus on the ongoing rewrite of the cephadm documentation and making certain that the documentation addresses the rough edges in the Pacific release. Meeting: https://bluejeans.com/908675367 Etherpad: https://pad.ceph.com/p/Ceph_Documentation

3 years, 1 month

1
0
0 0

Bluestore OSD crash with tcmalloc::allocate_full_cpp_throw_oom in multisite setup with PG_DAMAGED cluster error

by Szabo, Istvan (Agoda)

Hi, I have a 3 DC multisite setup. The replication is directional like HKG->SGP->US so the bucket is replicated from HKG to SGP and the same bucket is replicated further from SGP to US. The HKG > SGP connection is pretty fast 12.5millions objects (600GB) transferred in 6.5 hours. Once the OSD crashed in SGP, it stopped the complete chain replication and made PG_DAMAGED cluster error. The pg can be repaired but the sync never started back only bucket sync disable/enable helped. I got OSD crash also in HKG BUT in ASH. ASH no any error, the replication speed is 2 millions objects in 6.5 hours which is like 90GB of data. This is the crash of the osd: { "backtrace": [ "(()+0x12b20) [0x7f597d3fbb20]", "(gsignal()+0x10f) [0x7f597c0667ff]", "(abort()+0x127) [0x7f597c050c35]", "(()+0x9009b) [0x7f597ca1c09b]", "(()+0x9653c) [0x7f597ca2253c]", "(()+0x96597) [0x7f597ca22597]", "(()+0x967f8) [0x7f597ca227f8]", "(()+0x19d24) [0x7f597e168d24]", "(tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x146) [0x7f597e18b0d6]", "(rocksdb::Arena::AllocateNewBlock(unsigned long)+0x43) [0x5632f08ccb93]", "(rocksdb::Arena::AllocateFallback(unsigned long, bool)+0x4b) [0x5632f08ccc3b]", "(rocksdb::ConcurrentArena::AllocateAligned(unsigned long, unsigned long, rocksdb::Logger*)+0xb4) [0x5632f07fae94]", "(()+0x1103e7e) [0x5632f085ae7e]", "(rocksdb::MemTable::Add(unsigned long, rocksdb::ValueType, rocksdb::Slice const&, rocksdb::Slice const&, bool, rocksdb::MemTablePostProcessInfo*)+0xcf) [0x5632f07f6f8f]", "(rocksdb::MemTableInserter::PutCFImpl(unsigned int, rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::ValueType)+0x452) [0x5632f08520e2]", "(rocksdb::MemTableInserter::PutCF(unsigned int, rocksdb::Slice const&, rocksdb::Slice const&)+0x17) [0x5632f0852e97]", "(rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler*) const+0x480) [0x5632f084ac20]", "(rocksdb::WriteBatchInternal::InsertInto(rocksdb::WriteThread::WriteGroup&, unsigned long, rocksdb::ColumnFamilyMemTables*, rocksdb::FlushScheduler*, bool, unsigned long, rocksdb::DB*, bool, bool, bool)+0x149) [0x5632f084ebe9]", "(rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1acd) [0x5632f078a03d]", "(rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x21) [0x5632f078ac11]", "(RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x8c) [0x5632f074180c]", "(RocksDBStore::submit_transaction(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x87) [0x5632f0742027]", "(BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)+0x426) [0x5632f0226376]", "(BlueStore::_kv_sync_thread()+0x176f) [0x5632f024bc1f]", "(BlueStore::KVSyncThread::entry()+0x11) [0x5632f0273791]", "(()+0x814a) [0x7f597d3f114a]", "(clone()+0x43) [0x7f597c12bf23]" ], "ceph_version": "15.2.9", "crash_id": "2021-03-04T14:55:45.094048Z_3d481fd3-7573-4cb7-9b22-20784b418e64", "entity_name": "osd.5", "os_id": "centos", "os_name": "CentOS Linux", "os_version": "8", "os_version_id": "8", "process_name": "ceph-osd", "stack_sig": "9643c370a20c0d34f5e8965ae4461e2a7cf709ab4183929239bc263d0e1eef94", "timestamp": "2021-03-04T14:55:45.094048Z", "utsname_hostname": "hostname", "utsname_machine": "x86_64", "utsname_release": "4.18.0-240.10.1.el8_3.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP Mon Jan 18 17:05:51 UTC 2021" } Any idea what I should tune? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 years, 1 month

2
1
0 0

2024

2023

2022

2021

2020

2019

ceph-users March 2021