August 2020 - ceph-users

Re: [doc] drivegroups advanced case
by Jan Fajerski 24 Aug '20

24 Aug '20

On Fri, Aug 21, 2020 at 06:52:19PM +0000, Tony Liu wrote: >Hi, > >Regarding to the YAML in this section, >https://ceph.readthedocs.io/en/latest/cephadm/drivegroups/#the-advanced-case > >Is the "rotational" supposed to be "1" meaning spinning HDD? Correct. This corresponds to /sys/block/<your block device>/queue/rotational. > > >Thanks! >Tony >_______________________________________________ >ceph-users mailing list -- ceph-users(a)ceph.io >To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer

1 0

OSD Crash, high RAM usage
by Cloud Guy 23 Aug '20

23 Aug '20

Hello TL;DR We have a Nautilus cluster which has been operating without issue for quite some time. Recently one OSD experienced a relatively slow and painful death. The OSD was purged (via dashboard), replaced and added as a new OSD (same ID). Upon rebuild, we notice the node hosting the replaced OSD to experience extraordinary memory consumption (upwards of 95%). Remaining nodes healthy, operating normally. No adverse performance observations within the cluster. Looking for community assistance in helping troubleshoot this anomaly and resolving. Detail - Ceph 14.2.11 - Originally Deployed using ceph-ansible 4.0 stable; - OSD nodes run CentOS 7, have 10 drives, 256G RAM, no swap; - Problematic OSD ramps memory consumption to 95-99% available RAM; - Problematic OSD node appears to ramp RAM utilization until OSDs crash - and restart – resembling a contiguous saw-tooth pattern / memory leak??; - Other OSD nodes average around 30-40G steady state consumption. Cluster Logs: 2020-08-23 08:36:59.458072 [INF] Cluster is now healthy 2020-08-23 08:36:59.458032 [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2274696/100424847 objects degraded (2.265%), 85 pgs degraded) 2020-08-23 08:36:56.352104 [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 4 pgs down, 2 pgs peering, 15 pgs incomplete) 2020-08-23 08:36:56.352062 [WRN] Health check update: Degraded data redundancy: 2320455/100424847 objects degraded (2.311%), 87 pgs degraded (PG_DEGRADED) 2020-08-23 08:36:55.401933 [INF] osd.16 [v2:10.1.10.7:6831/204520,v1:10.1.10.7:6833/204520] boot 2020-08-23 08:36:54.744590 [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2020-08-23 08:36:53.688833 [WRN] Health check update: 1 osds down (OSD_DOWN) 2020-08-23 08:36:49.714790 [WRN] Health check update: Degraded data redundancy: 6617521/100424847 objects degraded (6.590%), 298 pgs degraded (PG_DEGRADED) 2020-08-23 08:36:49.714769 [WRN] Health check update: Reduced data availability: 17 pgs inactive, 5 pgs down, 2 pgs peering, 41 pgs incomplete (PG_AVAILABILITY) 2020-08-23 08:36:49.701966 [INF] osd.32 [v2:10.1.10.7:6812/204477,v1:10.1.10.7:6813/204477] boot 2020-08-23 08:36:49.701868 [INF] osd.28 [v2:10.1.10.7:6820/204478,v1:10.1.10.7:6821/204478] boot 2020-08-23 08:36:48.613586 [INF] osd.24 [v2:10.1.10.7:6836/204523,v1:10.1.10.7:6837/204523] boot 2020-08-23 08:36:48.540225 [WRN] Health check update: 3 osds down (OSD_DOWN) 2020-08-23 08:36:43.755645 [WRN] Health check failed: Degraded data redundancy: 5700895/100424847 objects degraded (5.677%), 266 pgs degraded (PG_DEGRADED) 2020-08-23 08:36:43.755611 [WRN] Health check failed: Reduced data availability: 12 pgs inactive, 5 pgs down, 37 pgs incomplete (PG_AVAILABILITY) 2020-08-23 08:36:40.049275 [INF] osd.16 failed (root=default,host=otr2817e02stor23) (connection refused reported by osd.39) 2020-08-23 08:36:40.047058 [INF] osd.24 failed (root=default,host=otr2817e02stor23) (connection refused reported by osd.6) 2020-08-23 08:36:39.452024 [WRN] Health check failed: 2 osds down (OSD_DOWN) 2020-08-23 08:36:39.399920 [INF] osd.32 failed (root=default,host=otr2817e02stor23) (connection refused reported by osd.33) 2020-08-23 08:36:39.399255 [INF] osd.28 failed (root=default,host=otr2817e02stor23) (connection refused reported by osd.18) 2020-08-23 08:04:08.089438 [INF] Cluster is now healthy 2020-08-23 08:04:08.089399 [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1549004/100424847 objects degraded (1.542%), 62 pgs degraded) 2020-08-23 08:04:05.154839 [WRN] Health check update: Degraded data redundancy: 1873878/100424847 objects degraded (1.866%), 68 pgs degraded (PG_DEGRADED) 2020-08-23 08:04:02.728402 [INF] osd.24 [v2:10.1.10.7:6820/202960,v1:10.1.10.7:6821/202960] boot 2020-08-23 08:04:02.572426 [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2020-08-23 08:03:53.138693 [WRN] Health check update: Degraded data redundancy: 2507272/100424847 objects degraded (2.497%), 88 pgs degraded (PG_DEGRADED) 2020-08-23 08:03:48.049569 [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 59 pgs peering) 2020-08-23 08:03:45.170204 [WRN] Health check failed: Degraded data redundancy: 281695/100424847 objects degraded (0.281%), 15 pgs degraded (PG_DEGRADED) 2020-08-23 08:03:45.170175 [WRN] Health check failed: Reduced data availability: 7 pgs inactive, 59 pgs peering (PG_AVAILABILITY) 2020-08-23 08:03:41.355393 [WRN] Health check failed: 1 osds down (OSD_DOWN)

2 1

SEO Services Company
by kseoagency＠gmail.com 23 Aug '20

23 Aug '20

K SEO Agency an SEO Company in Bangalore focused on providing full-service Search Engine Optimization Solutions that include specialized Local SEO, National SEO, Ecommerce SEO services for businesses in Bangalore, India. https://kseo.agency/

1 0

seo for small business
by kseoagency＠gmail.com 23 Aug '20

23 Aug '20

K SEO Agency an SEO Company in Bangalore focused on providing full-service Search Engine Optimization Solutions that include specialized Local SEO, National SEO, Ecommerce SEO services for businesses in Bangalore, India. https://kseo.agency/

1 0

ISO Certification in Kolkata
by kseoagency＠gmail.com 23 Aug '20

23 Aug '20

ISO Certification in Kolkata – Genveritas a global ISO Certification Consulting firm represents considerable authority in tweaked and result-situated answers for assisting organizations to actualize change and improve business execution. https://genveritas.in/iso-certification-in-kolkata/

1 0

How to change wal block in bluestore?
by Xu Xiao 23 Aug '20

23 Aug '20

Hi, I want to change my wal device to another device. I search much time and write a shell script, but finally, when I start the osd, the process is crashed. I think something is wrong, please help me check the script. Thanks! ceph version: ceph version 14.2.9 (bed944f8c45b9c98485e99b70e11bbcec6f6659a) nautilus (stable). The shell script is below: #! /bin/bash id=$1 # osd id vg=$2 # vg name osdPath="/var/lib/ceph/osd/ceph-$id" osdService="ceph-osd(a)${id}.service" blockPath=$(ls -lh ${osdPath}/block|awk '{print $11}') if [ -z "$blockPath" ]; then echo "no block path" exit 1; fi walPath=$(ls -lh ${osdPath}/block.wal|awk '{print $11}') if [ -z "$walPath" ]; then echo "no wal path" exit 2; fi walSize=$(blockdev --getsize64 $walPath) if [ -z "$walSize" ]; then echo "no wal size" exit 3; fi walUUID=$(lvdisplay ${walPath}|grep "LV UUID"|awk '{print $3}') if [ -z "$walUUID" ]; then echo "no wal uuid" exit 3; fi newWalLv="journal-osd${id}" newWalPath="/dev/${vg}/${newWalLv}" lvcreate -n "${newWalLv}" -L${walSize}b $vg || exit 4 newWalUUID=$(lvdisplay ${newWalPath}|grep "LV UUID"|awk '{print $3}') if [ -z "$newWalUUID" ]; then echo "no new wal uuid, new wal path:${newWalPath}" exit 3; fi ceph osd set noout || exit 5 systemctl stop ${osdService} || exit 6 ceph-osd -i $id --flush-journal || exit 7 dd if=${walPath} of=${newWalPath} bs=4M || exit 8 chown -h ceph:ceph $(realpath ${newWalPath}) || exit 9 ceph-bluestore-tool bluefs-bdev-migrate --path ${osdPath} --devs-source ${osdPath}/block.wal --dev-target ${newWalPath} chown -h ceph:ceph ${osdPath}/block.wal || exit 11 # device lvchange --deltag "ceph.wal_device=${walPath}" ${blockPath} || exit 12 lvchange --addtag "ceph.wal_device=${newWalPath}" ${blockPath} || exit 13 # for uuid lvchange --deltag "ceph.wal_uuid=${walUUID}" ${blockPath} || exit 12 lvchange --addtag "ceph.wal_uuid=${newWalUUID}" ${blockPath} || exit 13 # for wal lvs for tag in $(lvs -o lv_tags ${blockPath}|grep ceph|sed 's/,/\n/g') do lvchange --addtag "$tag" ${newWalPath} done lvchange --deltag "ceph.type=block" ${newWalPath} lvchange --addtag "ceph.type=wal" ${newWalPath} systemctl start ceph-osd(a)${id}.service || exit 14 ceph osd unset noout || exit 15

1 0

Add OSD with primary on HDD, WAL and DB on SSD
by Tony Liu 23 Aug '20

23 Aug '20

Hi, I am able to add OSD with primary on HDD, WAL and DB on SSD on dashboard. LVs are created on SSD for OSDs. How can I achieve the same by CLI? I tried with drivegroups, which adds OSD, but not WAL and DB. Should I handle LV for WAL and DB separately? Anything I am missing here? Thanks! Tony

1 0

How to Find the Cheap & Best Assignment Help Sydney Available Online?
by maxwillor＠gmail.com 22 Aug '20

22 Aug '20

I have been a part of Myassignmenthelpau I am working as a fully functional academic consultant with the team. We offer affordable pricing solutions for assignments and homework across all course subjects. I hold excellent writing skills and offer Assignment help Sydney of the best form. https://www.myassignmenthelpau.com/assignment-help-sydney

1 0

does ceph RBD have the ability to load balance?
by 差很多先生 22 Aug '20

22 Aug '20

Hello Cephers, I am a newcomer to Ceph.I know that RBD is a distributed block storage device and libvirt supports rbd pool type. Suppose I have a ceph cluster and use it to build a rbd pool to store virtual machine images. When multiple qemu-kvm clients use the image in the rbd pool to start virtual machines at the same time, will rbd perform load balancing? That is, different hosts in the cluster would send images to different clients at the same time. Or only one host will send images, and the other hosts will only send images when the current host is down? Would someone explain it to me please? Thanks consteven

2 1

Get Instant HELP From Epson support For Epson Error Code 0x97.
by mary smith 22 Aug '20

22 Aug '20

The Epson Error Code 0x97 is a kind of caution at whatever point you experience an inside equipment issue with your printer. It very well may be a result of a motherboard glitch or some other sort of inner segment breakdown. You should choose the guide from Epson support , straightforwardly from comfort of your home. https://www.epsonprintersupportpro.net/epson-error-code-0x97/

1 0

[doc] drivegroups advanced case
by Tony Liu 21 Aug '20

21 Aug '20

Hi, Regarding to the YAML in this section, https://ceph.readthedocs.io/en/latest/cephadm/drivegroups/#the-advanced-case Is the "rotational" supposed to be "1" meaning spinning HDD? Thanks! Tony

1 0

Editing Crush Map to fix osd_crush_chooseleaf_type = 0
by Matt Dunavant 21 Aug '20

21 Aug '20

Hi all, We have a 12 OSD node cluster in which I just recently found out that 'osd_crush_chooseleaf_type = 0' made it's way into our ceph.conf file, probably from previous testing. I believe this is the reason a recent maintenance on an OSD node caused data to stop flowing. In researching how to fix this, I just wanted to confirm a few things and see if anybody who has done this before has any perspective or things to look out for. 1) I believe the correct way to fix this is by following the 5 step method in the documentation; Get, Decompile, Edit, Recompile, Set. Is that correct and is the line I should change 'choose_firstn' to 'chooseleaf_firstn'? Do I only make this change on 1 mon and it will propagate it to all other mons and osds? 2) Does the process start immediately following the setcrushmap command? 3) Any files to backup prior to this operation? This is production data so we can not have any data loss. 4) Any other notes/things to be aware of? Thank you

2 7

OSDs get full with bluestore logs
by Khodayar Doustar 21 Aug '20

21 Aug '20

Hi, I have a 3 node cluster of mimic with 9 osds (3 osds on each node). I use this cluster to test integration of an application with S3 api. The problem is that after a few days all OSD starts filling up with bluestore logs and goes down and out one by one! I cannot stop the logs and I cannot find the setting to fix this leakage, this should be a leakage in logs because it's not logical to fill up all OSD with bluefs logs. This is an example of logs which is being repeated in bluestore logs: [root@server2 ~]# ceph-bluestore-tool --command bluefs-log-dump --path /var/lib/ceph/osd/ceph-5 . . . 0x40d000: op_file_update file(ino 30 size 0x7a713 mtime 2020-04-18 15:54:29.056488 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) 0x40e000: txn(seq 1156100 len 0x78 crc 0x3e1c626f) 0x40e000: op_file_update file(ino 30 size 0x7a72a mtime 2020-04-18 15:54:30.057828 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) 0x40f000: txn(seq 1156101 len 0x78 crc 0xc1f9ec5f) 0x40f000: op_file_update file(ino 30 size 0x7a741 mtime 2020-04-18 15:54:31.059252 bdev 1 allocated 80000 extents [1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000]) *** Caught signal (Segmentation fault) ** in thread 7f4108cfe600 thread_name:ceph-bluestore- ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (()+0xf5f0) [0x7f40fd47f5f0] 2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x3bf) [0x55603a1ebb3f] 3: (BlueFS::_replay(bool, bool)+0x214) [0x55603a1f6654] 4: (BlueFS::log_dump(CephContext*, std::string const&, std::vector<std::string, std::allocator<std::string> > const&)+0x3b) [0x55603a1fad3b] 5: (log_dump(CephContext*, std::string const&, std::vector<std::string, std::allocator<std::string> > const&)+0x64) [0x55603a1da764] 6: (main()+0x2f3e) [0x55603a105e3e] 7: (__libc_start_main()+0xf5) [0x7f40fbe51505] 8: (()+0x23115f) [0x55603a1d915f] 2020-04-18 21:08:25.410 7f4108cfe600 -1 *** Caught signal (Segmentation fault) ** in thread 7f4108cfe600 thread_name:ceph-bluestore- . . . And this is the output of daemonperf for one of remaining (filling up!) OSDs: [root@server2 ~]# ceph daemonperf osd.5 ------bluefs------- ------------bluestore------------- ----------osd----------- jlen j wal sst |fl_l k_l io_l th_l s_l c_l r_l |ops wr rd l rop | 81M 0 0 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.8k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0 0 ^C[root@server2 ~]# And this is the enospc logs when trying to start OSD: [root@server2 ~]# journalctl -u -f ceph-osd(a)4.service Failed to add match 'ceph-osd(a)4.service': Invalid argument Failed to add filters: Invalid argument [root@server2 ~]# journalctl -f -u ceph-osd(a)4.service -- Logs begin at Sat 2020-04-18 15:38:15 +0430. -- Apr 18 21:18:22 server2 ceph-osd[17485]: 22: (()+0x378c10) [0x557348ebac10] Apr 18 21:18:22 server2 ceph-osd[17485]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:18:22 server2 systemd[1]: ceph-osd(a)4.service: main process exited, code=killed, status=6/ABRT Apr 18 21:18:22 server2 systemd[1]: Unit ceph-osd(a)4.service entered failed state. Apr 18 21:18:22 server2 systemd[1]: ceph-osd(a)4.service failed. Apr 18 21:18:42 server2 systemd[1]: ceph-osd(a)4.service holdoff time over, scheduling restart. Apr 18 21:18:42 server2 systemd[1]: Stopped Ceph object storage daemon osd.4. Apr 18 21:18:42 server2 systemd[1]: Starting Ceph object storage daemon osd.4... Apr 18 21:18:42 server2 systemd[1]: Started Ceph object storage daemon osd.4. Apr 18 21:18:43 server2 ceph-osd[17777]: starting osd.4 at - osd_data /var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x794de9 Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094 7f43c936bb80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289 Apr 18 21:24:37 server2 ceph-osd[17777]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1704: FAILED assert(0 == "bluefs enospc") Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f43c074987b] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099 7f43c936bb80 -1 *** Caught signal (Aborted) ** Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80 thread_name:ceph-osd Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0] Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37) [0x7f43bc6db337] Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28] Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x248) [0x7f43c0749978] Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07] Apr 18 21:24:37 server2 ceph-osd[17777]: 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1ac6) [0x55ad52029266] Apr 18 21:24:37 server2 ceph-osd[17777]: 7: (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd] Apr 18 21:24:37 server2 ceph-osd[17777]: 8: (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916] Apr 18 21:24:37 server2 ceph-osd[17777]: 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde] Apr 18 21:24:37 server2 ceph-osd[17777]: 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8] Apr 18 21:24:37 server2 ceph-osd[17777]: 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76] Apr 18 21:24:37 server2 ceph-osd[17777]: 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b) [0x55ad520b2dcb] Apr 18 21:24:37 server2 ceph-osd[17777]: 13: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xa59) [0x55ad520b3d09] Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x689) [0x55ad520b4ab9] Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0x55ad520b62e2] Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de] Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool, bool)+0xcf4) [0x55ad51f527a4] Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool, bool)+0x4e9) [0x55ad51f828f9] Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339) [0x55ad51b1cf09] Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2) [0x55ad51a00d52] Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5) [0x7f43bc6c7505] Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10] Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Apr 18 21:24:37 server2 systemd[1]: ceph-osd(a)4.service: main process exited, code=killed, status=6/ABRT Apr 18 21:24:37 server2 systemd[1]: Unit ceph-osd(a)4.service entered failed state. Apr 18 21:24:37 server2 systemd[1]: ceph-osd(a)4.service failed. Apr 18 21:24:57 server2 systemd[1]: ceph-osd(a)4.service holdoff time over, scheduling restart. Apr 18 21:24:57 server2 systemd[1]: Stopped Ceph object storage daemon osd.4. Apr 18 21:24:57 server2 systemd[1]: Starting Ceph object storage daemon osd.4... Apr 18 21:24:57 server2 systemd[1]: Started Ceph object storage daemon osd.4. and some more data: [root@server1 ~]# ceph df detail GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 38 GiB 35 GiB 2.6 GiB 6.83 303 POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED .rgw.root 18 N/A N/A 1.1 KiB 0 50 GiB 4 4 63 B 4 B 1.1 KiB default.rgw.meta 19 N/A N/A 4.1 KiB 0 50 GiB 22 22 1.5 KiB 182 B 9.0 KiB default.rgw.log 20 N/A N/A 0 B 0 50 GiB 107 107 307 KiB 205 KiB 0 B default.rgw.control 21 N/A N/A 0 B 0 50 GiB 5 5 0 B 0 B 0 B default.rgw.buckets.index 22 N/A N/A 0 B 0 50 GiB 5 5 501 B 60 B 0 B default.rgw.buckets.data 23 N/A N/A 377 MiB 0.39 50 GiB 160 160 385 B 1.1 KiB 603 MiB [root@server1 ~]# [root@server1 ~]# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS TYPE NAME -1 0.16727 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - root default -3 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host server1 0 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.0 1 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.1 2 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.2 -5 0.05576 - 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 0 0 - host server2 3 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.3 4 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.4 5 hdd 0.01859 1.00000 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 7.11 1.04 99 osd.5 -7 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host server3 6 hdd 0.01859 1.00000 19 GiB 1.2 GiB 249 MiB 3 KiB 1024 MiB 18 GiB 6.55 0.96 78 osd.6 7 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.7 8 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 osd.8 TOTAL 38 GiB 2.6 GiB 610 MiB 6 KiB 2.0 GiB 35 GiB 6.83 MIN/MAX VAR: 0/1.04 STDDEV: 5.58 [root@server1 ~]# I'm kind of newbie to ceph, so any help or hint would be appreciated. Did I hit a bug or something is wrong with my configuration? Thanks a lot, Khodayar

5 5

BlueFS spillover detected, why, what?
by Simon Oosthoek 21 Aug '20

21 Aug '20

Hi Recently our ceph cluster (nautilus) is experiencing bluefs spillovers, just 2 osd's and I disabled the warning for these osds. (ceph config set osd.125 bluestore_warn_on_bluefs_spillover false) I'm wondering what causes this and how this can be prevented. As I understand it the rocksdb for the OSD needs to store more than fits on the NVME logical volume (123G for 12T OSD). A way to fix it could be to increase the logical volume on the nvme (if there was space on the nvme, which there isn't at the moment). This is the current size of the cluster and how much is free: [root@cephmon1 ~]# ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.8 PiB 842 TiB 974 TiB 974 TiB 53.63 TOTAL 1.8 PiB 842 TiB 974 TiB 974 TiB 53.63 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 572 MiB 121.26M 2.4 GiB 0 167 TiB cephfs_metadata 2 56 GiB 5.15M 57 GiB 0 167 TiB cephfs_data_3copy 8 201 GiB 51.68k 602 GiB 0.09 222 TiB cephfs_data_ec83 13 643 TiB 279.75M 953 TiB 58.86 485 TiB rbd 14 21 GiB 5.66k 64 GiB 0 222 TiB .rgw.root 15 1.2 KiB 4 1 MiB 0 167 TiB default.rgw.control 16 0 B 8 0 B 0 167 TiB default.rgw.meta 17 765 B 4 1 MiB 0 167 TiB default.rgw.log 18 0 B 207 0 B 0 167 TiB cephfs_data_ec57 20 433 MiB 230 1.2 GiB 0 278 TiB The amount used can still grow a bit before we need to add nodes, but apparently we are running into the limits of our rocskdb partitions. Did we choose a parameter (e.g. minimal object size) too small, so we have too much objects on these spillover OSDs? Or is it that too many small files are stored on the cephfs filesystems? When we expand the cluster, we can choose larger nvme devices to allow larger rocksdb partitions, but is that the right way to deal with this, or should we adjust some parameters on the cluster that will reduce the rocksdb size? Cheers /Simon

4 15

How do I configure a Roadrunner email account?
by smith8395john＠gmail.com 21 Aug '20

21 Aug '20

I am a professional person and dealing with small and big customers. So I need an emailing application to communicate with them online. On a daily basis, I have to make many business deals with the customers. So I need to send business deals details. So I have decided to set up a roadrunner email account on my laptop. Roadrunner email service is an admirable emailing application, which is mainly used by the countless users. It is mainly used by both home and business users. I am also a small business person and look for choosing roadrunner email service for my daily emailing communication. Due to a lack of technical knowledge, I am not able to configure a roadrunner email account on my computer system. I don’t have complete technical knowledge about the roadrunner email settings. As per my technical knowledge, I am applying all right things for the account settings, even though I am failing to set up the roadrunner email account successfully. When I try to set up a roadrunner email account on my computer system, I am experiencing technical difficulty for the configuration process of the roadrunner email account. So anyone can recommend the simple ways to set up a roadrunner email account correctly. https://www.emailsupport.us/blog/configure-roadrunner-email-settings

1 0

Remove Error - "Possible data damage: 2 pgs recovery_unfound"
by Jonathan Sélea 21 Aug '20

21 Aug '20

Good afternoon! I have a small Ceph-cluster running with Proxmox, and after an update on one of the nodes and a reboot. So far so good. But after a couple of hours, I saw this: root@pve2:~# ceph health detail HEALTH_ERR 16/1101836 objects unfound (0.001%); Possible data damage: 2 pgs recovery_unfound; Degraded data redundancy: 48/3305508 objects degraded (0.001%), 2 pgs degraded, 2 pgs undersized OBJECT_UNFOUND 16/1101836 objects unfound (0.001%) pg 1.37 has 6 unfound objects pg 1.48 has 10 unfound objects PG_DAMAGED Possible data damage: 2 pgs recovery_unfound pg 1.37 is active+recovery_unfound+undersized+degraded+remapped, acting [11,17], 6 unfound pg 1.48 is active+recovery_unfound+undersized+degraded+remapped, acting [5,11], 10 unfound PG_DEGRADED Degraded data redundancy: 48/3305508 objects degraded (0.001%), 2 pgs degraded, 2 pgs undersized pg 1.37 is stuck undersized for 446774.454853, current state active+recovery_unfound+undersized+degraded+remapped, last acting [11,17] pg 1.48 is stuck undersized for 446774.459466, current state active+recovery_unfound+undersized+degraded+remapped, last acting [5,11] root@pve2:~# ceph -s cluster: id: 76e70c34-bce9-4f86-b049-0054f21c3494 health: HEALTH_ERR 16/1101836 objects unfound (0.001%) Possible data damage: 2 pgs recovery_unfound Degraded data redundancy: 48/3305508 objects degraded (0.001%), 2 pgs degraded, 2 pgs undersized services: mon: 3 daemons, quorum pve3,pve1,pve2 (age 2w) mgr: pve3(active, since 2w), standbys: pve1, pve2 mds: cephfs:1 {0=pve1=up:active} 2 up:standby osd: 25 osds: 25 up (since 5d), 25 in (since 8d); 2 remapped pgs data: pools: 4 pools, 672 pgs objects: 1.10M objects, 2.9 TiB usage: 8.6 TiB used, 12 TiB / 21 TiB avail pgs: 48/3305508 objects degraded (0.001%) 16/1101836 objects unfound (0.001%) 669 active+clean 2 active+recovery_unfound+undersized+degraded+remapped 1 active+clean+scrubbing+deep io: client: 680 B/s rd, 2.6 MiB/s wr, 0 op/s rd, 151 op/s wr I am not really concerned over lost data, since I am 99% sure it belonged to a faulty prometheus server anyway. The question is, how can I remove the warnings without affecting the other objects? Thankful for any pointers! -- Jonathan Sélea Website: https://jonathanselea.se PGP Key: 0x8B35B3C894B964DD Fingerprint: 4AF2 10DE 996B 673C 0FD8 AFA0 8B35 B3C8 94B9 64DD

2 2

5 pgs inactive, 5 pgs incomplete
by Martin Palma 21 Aug '20

21 Aug '20

Hello, after an unexpected power outage our production cluster has 5 PGs inactive and incomplete. The OSDs on which these 5 PGs are located all show "stuck requests are blocked": Reduced data availability: 5 pgs inactive, 5 pgs incomplete 98 stuck requests are blocked > 4096 sec. Implicated osds 63,80,492,494 What is the best procedure to get these PGs back? These PGs are all of pools with a replica of 2. Best, Martin

7 17

Bonus Ceph Tech Talk: Edge Application - Stream Multiple Video Sources
by Mike Perez 20 Aug '20

20 Aug '20

Hi all, We have a bonus Ceph Tech Talk for August. Join us August 20th at 17:00 UTC to hear Neeha Kompala and Jason Weng present on Edge Application - Streaming Multiple Video Sources. Don't forget on August 27th at 17:00 UTC, Pritha Srivastava will also be presenting on this month's Ceph Tech Talk: Secure Token Service in the Rados Gateway. If you're interested in giving a Ceph Tech Talk for September 24th or October 22nd, please let me know! https://ceph.io/ceph-tech-talks/ -- Mike Perez He/Him Ceph Community Manager Red Hat Los Angeles <https://www.redhat.com> thingee(a)redhat.com <mailto:thingee@redhat.com> M: 1-951-572-2633 <tel:1-951-572-2633> IM: IRC Freenode/OFTC: thingee 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> <https://www.redhat.com> <https://redhat.com/summit>

3 4

Re: Bonus Ceph Tech Talk: Edge Application - Stream Multiple Video Sources
by Bobby 20 Aug '20

20 Aug '20

Hi...Will it be available on youtube? On Thursday, August 20, 2020, Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > Can't join as guest without enabling mic and/or camera??? > > -----Original Message----- > From: Mike Perez [mailto:miperez@redhat.com] > Sent: donderdag 20 augustus 2020 19:03 > To: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Bonus Ceph Tech Talk: Edge Application - > Stream Multiple Video Sources > > And we're live! Please join us and bring questions! > > https://bluejeans.com/908675367 > > On 8/17/20 11:03 AM, Mike Perez wrote: >> >> Hi all, >> >> We have a bonus Ceph Tech Talk for August. Join us August 20th at >> 17:00 UTC to hear Neeha Kompala and Jason Weng present on Edge >> Application - Streaming Multiple Video Sources. >> >> Don't forget on August 27th at 17:00 UTC, Pritha Srivastava will also >> be presenting on this month's Ceph Tech Talk: Secure Token Service in >> the Rados Gateway. >> >> If you're interested in giving a Ceph Tech Talk for September 24th or >> October 22nd, please let me know! >> >> https://ceph.io/ceph-tech-talks/ >> >> -- >> >> Mike Perez >> >> He/Him >> >> Ceph Community Manager >> >> Red Hat Los Angeles <https://www.redhat.com> >> >> thingee(a)redhat.com <mailto:thingee@redhat.com> >> M: 1-951-572-2633 <tel:1-951-572-2633> IM: IRC Freenode/OFTC: thingee >> >> 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA >> >> @Thingee <https://twitter.com/thingee> >> <https://www.redhat.com> >> <https://redhat.com/summit> >> > -- > > Mike Perez > > He/Him > > Ceph Community Manager > > Red Hat Los Angeles <https://www.redhat.com> > > thingee(a)redhat.com <mailto:thingee@redhat.com> > M: 1-951-572-2633 <tel:1-951-572-2633> IM: IRC Freenode/OFTC: thingee > > 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA > > @Thingee <https://twitter.com/thingee> > <https://www.redhat.com> > <https://redhat.com/summit> > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

radosgw beast access logs
by Graham Allan 20 Aug '20

20 Aug '20

Are there any plans to add access logs to the beast frontend, in the same way we can get with civetweb? Increasing the "debug rgw" setting really doesn't provide the same thing. Graham -- Graham Allan - gta(a)umn.edu Associate Director of Operations - Minnesota Supercomputing Institute

4 4

luks / disk encryption best practice
by Marc Roos 20 Aug '20

20 Aug '20

I still need to move from ceph disk to ceph volume. When doing this, I wanted to also start using disk encryption. I am not really interested in encryption offered by the hdd vendors. Is there a best practice or advice what encryption to use ciphers/hash? Stick to the default of CentOS7 or maybe choose what is default in CentOS or something else? Different settings for ssd / hdd?

1 0

Ceph on windows?
by Stolte, Felix 20 Aug '20

20 Aug '20

Hey guys, it seems like there was a presentation called “ceph on windows” at the Cephalocon 2020, but I cannot find any information on that topic. Is there a video from the presentation out there or any other information? I only found https://ceph2020.sched.com/event/ZDUK/ceph-on-windows-alessandro-pilotti-cl… Would be great to mount rbds directly instead using an iscsi gateway. IT-Services Telefon 02461 61-9243 E-Mail: f.stolte(a)fz-juelich.de ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------

3 2

Ceph mon crash, many osd down
by hoannv46＠gmail.com 20 Aug '20

20 Aug '20

Hi all. My cluster has many log mon scrub mon data 2020-08-20 13:12:16.393 7fe89becc700 0 log_channel(cluster) log [DBG] : scrub ok on 0,1,2,3: ScrubResult(keys {auth=100} crc {auth=3066031631}) 2020-08-20 13:12:16.395 7fe89becc700 0 log_channel(cluster) log [DBG] : scrub ok on 0,1,2,3: ScrubResult(keys {auth=100} crc {auth=221313478}) 2020-08-20 13:12:16.401 7fe89becc700 0 log_channel(cluster) log [DBG] : scrub ok on 0,1,2,3: ScrubResult(keys {auth=15,config=2,health=10,logm=73} crc {auth=2119885989,config=3307175017,health=67914304,logm=3854202346}) 2020-08-20 13:12:16.404 7fe89becc700 0 log_channel(cluster) log [DBG] : scrub ok on 0,1,2,3: ScrubResult(keys {logm=100} crc {logm=3116621380}) 2020-08-20 13:12:16.408 7fe89becc700 0 log_channel(cluster) log [DBG] : scrub ok on 0,1,2,3: ScrubResult(keys {logm=100} crc {logm=767596958}) 2020-08-20 13:12:16.411 7fe89becc700 0 log_channel(cluster) log [DBG] : scrub ok on 0,1,2,3: ScrubResult(keys {logm=100} crc {logm=3982727178}) 2020-08-20 13:12:16.414 7fe89becc700 0 log_channel(cluster) log [DBG] : scrub ok on 0,1,2,3: ScrubResult(keys {logm=100} crc {logm=4144183080}) after 900 seconds, mon mark osd down 2020-08-20 13:23:32.546 7fe89e6d1700 0 log_channel(cluster) log [INF] : osd.112 marked down after no beacon for 904.106586 seconds 2020-08-20 13:23:32.546 7fe89e6d1700 -1 mon.ceph-mon-1(a)0(leader).osd e2960665 no beacon from osd.112 since 2020-08-20 13:08:28.441052, 904.106586 seconds ago. marking down 2020-08-20 13:23:32.551 7fe89e6d1700 0 log_channel(cluster) log [WRN] : Health check failed: 1 osds down (OSD_DOWN) 2020-08-20 13:24:07.899 7fe89e6d1700 0 log_channel(cluster) log [INF] : osd.263 marked down after no beacon for 901.445447 seconds 2020-08-20 13:24:07.899 7fe89e6d1700 -1 mon.ceph-mon-1(a)0(leader).osd e2960666 no beacon from osd.263 since 2020-08-20 13:09:06.454891, 901.445447 seconds ago. marking down 2020-08-20 13:24:07.902 7fe89e6d1700 0 log_channel(cluster) log [WRN] : Health check update: 2 osds down (OSD_DOWN) 2020-08-20 13:24:13.020 7fe89e6d1700 0 log_channel(cluster) log [INF] : osd.384 marked down after no beacon for 900.132560 seconds 2020-08-20 13:24:13.020 7fe89e6d1700 -1 mon.ceph-mon-1(a)0(leader).osd e2960667 no beacon from osd.384 since 2020-08-20 13:09:12.888844, 900.132560 seconds ago. marking down 2020-08-20 13:24:13.020 7fe89e6d1700 0 log_channel(cluster) log [INF] : osd.614 marked down after no beacon for 901.359447 seconds 2020-08-20 13:24:13.020 7fe89e6d1700 -1 mon.ceph-mon-1(a)0(leader).osd e2960667 no beacon from osd.614 since 2020-08-20 13:09:11.661958, 901.359447 seconds ago. marking down 2020-08-20 13:24:13.026 7fe89e6d1700 0 log_channel(cluster) log [WRN] : Health check update: 4 osds down (OSD_DOWN) 2020-08-20 13:24:18.084 7fe89e6d1700 0 log_channel(cluster) log [INF] : osd.34 marked down after no beacon for 903.818250 seconds is this bug of ceph mon. Thanks.

1 0

cephadm not working with non-root user
by Amudhan P 20 Aug '20

20 Aug '20

Hi, I am trying to install ceph 'octopus' using cephadm. In bootstrap command, I have specified a non-root user account as ssh-user. cephadm bootstrap --mon-ip xx.xxx.xx.xx --ssh-user non-rootuser when bootstrap about to complete it threw an error stating. """" INFO:cephadm:Non-zero exit code 2 from /usr/bin/podman run --rm --net=host --ipc=host -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=node1 - v /var/log/ceph/ae4ed114-e145-11ea-9c1f-0025900a8ebe:/var/log/ceph:z -v /tmp/ceph-tmpm22k9j9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpe 1ltigk8:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph docker.io/ceph/ceph:v15 orch host add node1 INFO:cephadm:/usr/bin/ceph:stderr Error ENOENT: Failed to connect to node1 (node1). INFO:cephadm:/usr/bin/ceph:stderr Check that the host is reachable and accepts connections using the cephadm SSH key INFO:cephadm:/usr/bin/ceph:stderr INFO:cephadm:/usr/bin/ceph:stderr you may want to run: INFO:cephadm:/usr/bin/ceph:stderr > ceph cephadm get-ssh-config > ssh_config INFO:cephadm:/usr/bin/ceph:stderr > ceph config-key get mgr/cephadm/ssh_identity_key > key INFO:cephadm:/usr/bin/ceph:stderr > ssh -F ssh_config -i key root@node1 """"" In the above steps, it's trying to connect as root to the node and when I downloaded ssh_config file it was also specified as 'root' inside. so, I modified the config file and uploaded but same to ceph but still ssh to node1 is not working. To confirm if I have used the right command been used during bootstrap. I have tried the below command. " ceph config-key dump mgr/cephadm/ssh_user" { "mgr/cephadm/ssh_user": "non-rootuser" } and the output shows the user I have used during bootstrap "non-rootuser" but at the same time when I run cmd " ceph cephadm get-user " the output still shows 'root' as the user. Why the change is not affecting? do anyone faced a similar issue in bootstrap? Is there any way to avoid using container with cephadm? regards Amudhan

1 1

Ceph Snapshot Children not exists / children relation broken
by Torsten Ennenbach 20 Aug '20

20 Aug '20

Hello all, I`ve a problem with an undeletable Image. Let my try to explain. There is a storage with a Snapshot, and the snapshot thinks he has a Children: rbd snap unprotect delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [f907bc6b8b4567] in pool 'rbd' 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy 2020-07-30 09:25:08.492 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80 should_complete_error: ret_val=-16 2020-07-30 09:25:08.496 7f15f77fe700 -1 librbd::SnapshotUnprotectRequest: 0x559ca6dd7b80 should_complete_error: ret_val=-16rbd: unprotecting snap failed: (16) Device or resource busy rbd children delete-me-please@995cc2e3-c636-4c43-87c3-dbc729173c09 2020-07-30 09:19:36.650 7f39563deb00 -1 librbd::api::Image: list_descendants: error looking up name for image id f907bc6b8b4567 in pool rbd rbd: listing children failed: (2) No such file or directory So i am not able to unprotect and delete the Snapshot. Is there a way to fix this issue? I know this happens if deep-flatten is not enabled, and this is the root cause I think. We`ve activated this feature now but this s artifact from a time where it was disabled. Best regards & thx for your help. Torsten

3 8

CEPH FS is always showing the status as creating
by Alokkumar Mahajan 20 Aug '20

20 Aug '20

Hello, We have created CEPH FS but it is always showing the status as creating. ceph fs get returns below output:- =========================================== Filesystem 'cephfs' (4) fs_name cephfs epoch 2865929 flags 12 created 2020-08-07 05:05:58.033824 modified 2020-08-14 03:15:49.727680 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 min_compat_client -1 (unspecified) last_failure 0 last_failure_osd_epoch 652 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0 up {0=1494099} failed damaged stopped data_pools [32] metadata_pool 33 inline_data disabled balancer standby_count_wanted 0 1494099: 10.18.97.47:6800/2041780514 <http://10.128.97.247:6800/2041780514> 'vl-pun-qa' mds.0.2748022 up:creating seq 149289 ==================================================== We are CEPH 13.2.6 MIMIC Version. I am new to CEPH so i am really not sure where to start checking this, any help will be greatly appreciated. Thanks, -alok

3 3

When do I need to update my GPS maps?
by jacknick1996＠gmail.com 20 Aug '20

20 Aug '20

I am a traveler and using GPS technology for finding the unknown locations or places. I am using this technology from the last three months, so I cannot say that I have complete knowledge about Garmin GPS technology. I know that I should update my Garmin GPS unit on a regular basis. I don’t know when I need to update my Garmin GPS unit. Due to insufficient knowledge, I am not able to update Garmin GPS unit. Really, it is a very difficult job for me for Garmin updates. Therefore anyone can recommend the best ways to update my Garmin GPS maps. Read More:- https://www.garmingpsmap-update.com/

1 0

When do I need to update my GPS maps?
by jacknick1996＠gmail.com 20 Aug '20

20 Aug '20

I am a traveler and using GPS technology for finding the unknown locations or places. I am using this technology from the last three months, so I cannot say that I have complete knowledge about Garmin GPS technology. I know that I should update my Garmin GPS unit on a regular basis. I don’t know when I need to update my Garmin GPS unit. Due to insufficient knowledge, I am not able to update Garmin GPS unit. Really, it is a very difficult job for me for Garmin updates. Therefore anyone can recommend the best ways to update my Garmin GPS maps. Read More:-https://www.garmingpsmap-update.com/

1 0

When do I need to update my GPS maps?
by jacknick1996＠gmail.com 20 Aug '20

20 Aug '20

I am a traveler and using GPS technology for finding the unknown locations or places. I am using this technology from the last three months, so I cannot say that I have complete knowledge about Garmin GPS technology. I know that I should update my Garmin GPS unit on a regular basis. I don’t know when I need to update my Garmin GPS unit. Due to insufficient knowledge, I am not able to update Garmin GPS unit. Really, it is a very difficult job for me for [url=https://www.garmingpsmap-update.com/]Garmin updates[/url]. Therefore anyone can recommend the best ways to update my Garmin GPS maps.

1 0

Re: RGW Lifecycle Processing and Promote Master Process
by Casey Bodley 19 Aug '20

19 Aug '20

On Fri, Aug 14, 2020 at 9:25 AM Alex Hussein-Kershaw <Alex.Hussein-Kershaw(a)metaswitch.com> wrote: > > Hi, > > I've previously discussed some issues I've had with the RGW lifecycle processing. I've discovered that the root cause of my problem is that: > > * I'm running a multisite configuration > * Life cycle processing is done on the master site each night. `radosgw-admin lc list` correctly returns all buckets with lc config. > * I simulate the master site being destroyed from my VM host. > * I promote the secondary site to master following the instructions here: https://docs.ceph.com/docs/master/radosgw/multisite/ > * The new master site isn't doing any lifecycle processing. `radosgw-admin lc list` returns empty. > * I recreate a cluster and pair it with the new master site to get back to having multisite redundancy. > * Neither site is doing any lifecycle processing. `radosgw-admin lc list` returns empty. > So in the process of failover/recovery I have gone from having two paired clusters performing lifecycle processing, to two paired clusters NOT performing lifecycle processing. > > Is this behaviour expected? I've found `radosgw-admin lc reshard fix` will "remind" the cluster that I run it on that it needs to do lifecycle processing. Although I found no mention of having to use this in the docs, for that command the docs state it's only relevant on earlier Ceph versions. I'm running Nautilus 14.2.9. > > In addition, if I have two healthy clusters paired in a multisite system, and swap the master cluster by promoting the non-master, the demoted cluster seems to still continue doing lifecycle processing, while the promote does not. If I run `radosgw-admin lc reshard fix` on the promoted cluster, then both clusters seem to claim they are doing the processing. Is this a happy state to be in? > > Does anyone have any experience with this? > > Thanks, > Alex > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > There's a defect in metadata sync (https://tracker.ceph.com/issues/44268) which prevents buckets with lifecycle policies from being indexed for lifecycle processing on non-master zones. It sounds like the 'lc reshard fix' command is adding it back to that index for processing. The intent is for lifecycle processing to occur independently on every zone. That's the only way to guarantee the correct result now that we have PutBucketReplication (and specifically the Filter policy) where any given zone may only hold a subset of the objects from its source.

2 1

pubsub RGW and OSD processes suddenly start using much more CPU
by david.piper＠metaswitch.com 19 Aug '20

19 Aug '20

Hi all, I've got a 3-node cluster in a lab environment running on ceph version 14.2.9 (containerized). Each node is running a OSD, MON, MGR, MDS and 2 x RGW (we're using a second RGW instance to host a pubsub endpoint). I've been monitoring my nodes with the ceph dashboard, and noticed that all of a sudden at random the CPU profile of my hosts completely changes from [iowait: 20%, user: 10%, system: 5%, others: <5%] to [user: 40%, system: 20%, iowait: 5%, others: <5%]. This isn't a spike; the CPU profile is still like this after several days. All hosts seem to be affected at the same moment. Running `top` on the hosts shows it's the OSD and one of the RGW processes (seems to be the pubsub process) that are using all the CPU: $ top top - 14:57:23 up 11 days, 22:26, 1 user, load average: 1.65, 1.81, 2.04 Tasks: 140 total, 2 running, 138 sleeping, 0 stopped, 0 zombie %Cpu(s): 40.9 us, 21.4 sy, 0.0 ni, 27.9 id, 5.9 wa, 0.0 hi, 4.0 si, 0.0 st KiB Mem : 12304036 total, 183596 free, 10848600 used, 1271840 buff/cache KiB Swap: 0 total, 0 free, 0 used. 1184676 avail Mem Change delay from 3.0 to PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6541 167 20 0 5505324 205564 1380 S 67.8 1.7 4651:41 radosgw 3826 167 20 0 9563536 7.9g 5496 S 64.8 67.6 4609:05 ceph-osd 6621 167 20 0 5571764 276804 0 S 6.6 2.2 4985:30 radosgw 5524 167 20 0 1365368 322240 8516 S 3.7 2.6 171:37.23 ceph-mgr 1413 root 20 0 558728 28372 24968 S 0.3 0.2 18:56.05 rsyslogd .... If I restart the RGW instances (I think it's the pubsub instance specifically), CPU usage returns back to a normal level. I don't see any unusual logs (from the either RGW process or the OSD or MON processes) to indicate that anything has materially changed at the point the CPU elevates. Has anyone experienced anything like this? Beyond turning up RGW logging and trying to make sense of the output, I'm not sure where next to dig. Cheers, Dave

1 0

Convert existing rbd into a cinder volume
by Stolte, Felix 19 Aug '20

19 Aug '20

Hello fellow cephers, i know this is not the openstack mailing list, but since many of us are using ceph as a backend for openstack, maybe someone can help me out. I have a pool for rbds which are exported via iscsi to some bare metal windows servers and do snaphots on that rbds regulary. Now I want to clone a snapshot and attach the clone to an openstack instance (for restore purpose). I got it working, but the process is very uncomfortable and maybe someone can point me to a faster way. For now it looks like the following: 1. Create the clone via 'rbd clone <pool>/<image>@<snapshotname> <pool>/<clonename> 2. Import the clone into the openstack admin project with 'cinder manage --name=<my_volume_name> <cinderhost>@<pool>#<pool> 3. Create a volume transfer request in the admin project 4. Accept the transfer request in the desired project 5. Attach the volume to the instance There has to be an easier and faster way to accomplish this. Has someone a better solution? Best regards Felix ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------

2 1

OSD takes more almost two hours to boot from Luminous -> Nautilus
by Mark Schouten 19 Aug '20

19 Aug '20

Hi, Last night I upgraded a Luminous cluster to Nautilus. All went well, but there was one sleep depriving issue I would like to prevent from happening next week while upgrading another cluster. Maybe you people can help me figure out what actually happened. So I upgraded the packages and restarted mons and mgrs. Then I started restarting the OSD's on one of the nodes. Below are the start and 'start_boot' times, during which the disks read at full speed, I think the whole disk. 2020-08-19 02:08:10.568 7fd742b09c80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 02:09:33.591 7fd742b09c80 1 osd.8 2188 start_boot 2020-08-19 02:08:10.592 7fb453887c80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 02:17:40.878 7fb453887c80 1 osd.5 2188 start_boot 2020-08-19 02:08:10.836 7f907bc0cc80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 02:19:58.462 7f907bc0cc80 1 osd.3 2188 start_boot 2020-08-19 02:08:10.584 7f1ca892cc80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 03:13:24.179 7f1ca892cc80 1 osd.11 2188 start_boot 2020-08-19 02:08:10.568 7f059f80dc80 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-08-19 04:06:55.342 7f059f80dc80 1 osd.14 2188 start_boot So, while this is not an issue which breaks anything technical, I would like to know how I can arrange for the OSD to do this 'maintenance' beforehand so I don't have to wait too long. :) I do see a warning in the logging, is that related: "store not yet converted to per-pool stats" ? Thanks! -- Mark Schouten <mark(a)tuxis.nl> Tuxis, Ede, https://www.tuxis.nl T: +31 318 200208

2 3

How To Reset HP Printer? Get In Touch With HP Support Assistant.
by mary smith 19 Aug '20

19 Aug '20

If you are not aware of the process of getting the real time process of resetting your HP printer, then you should fetch the technical backing directly from the HP Support Assistant. Here, you will be able to get the top-to-toe solutions under the supervision of the experts in a couple of seconds. https://www.amiytech.com/hp-support-assistant/

1 0

What to Do When Printer Goes Down? Contact Epson Customer Service.
by mary smith 19 Aug '20

19 Aug '20

Found your printer is down? Printer issues tend to frustrate users at times, especially when you need it the most. Now, you not need to panic. Epson Customer Service professionals can diagnose the problems and fix your printer problem in a matter of minutes. You just need to share your problems, the error code and other errors. They avail expertise and trained exclusively to provide instant services. https://www.epsonprintersupportpro.net/

1 0

Alpine linux librados-dev missing
by Marc Roos 18 Aug '20

18 Aug '20

I am not sure if I should try this, but I was trying to build this dovecot-ceph-plugin plugin on alpine linux to create a nice small container image. However alpine linux does not seem to have librados-dev. Did any one do something similar, and have a workaround for this?

1 0

why ceph-fuse init Objecter with osd_timeout = 0
by Ch Wan 18 Aug '20

18 Aug '20

Hi all I'm using the mimic-13.2.4 version and ceph-fuse as client. Recently I'm suffering a strange problem: on client machine we can see the tcp has ESTAB state to osd machine, but on osd machine nothing could be found. Then the client hangs on read/write requests from this osd So I'm trying to figure out why this happens, and search for configuration to set a timeout for osd requests. I noticed that osdc/Objecter.h has a osd_timeout field, which will be initiated when we create an Objecter. But in ceph-fuse, it creates an Objecter using a fixed value 0, meaning no timeout? StandaloneClient::StandaloneClient(Messenger *m, MonClient *mc, > boost::asio::io_context& ictx) > : Client(m, mc, new Objecter(m->cct, m, mc, ictx, 0, 0)) > Here are my questions: 1. Why ceph-fuse set osd_timeout to 0? 2. Do we have other configurations to let osd requests fail instead of hanging forever?

1 0

fio rados ioengine
by Frank Ritchie 18 Aug '20

18 Aug '20

Hi all, When testing with the fio rados ioengine is it necessary to run a write test with the no-cleanup option before running read tests like is required with rados bench? thx Frank

1 0

Epson Printer Printing With Lines Contact Epson Customer Service.
by mary smith 18 Aug '20

18 Aug '20

You are printing structure Epson printer and getting lines on the print, to determine this issue contact Epson Customer Service. On the off chance that you are not happy with the data gave on the web, don't hesitate to get in touch with us. We are consistently there to keep you grinning. You can request help at any second as we are accessible all day, every day. Kindly don't stop for a second; we have a committed proficient to assist you with time. https://www.epsonprintersupportpro.net/

1 0

Re: New ceph cluster - cephx disabled, now without access
by Eugen Block 18 Aug '20

18 Aug '20

Have you tried restarting the ceph services? If those config options are not in ceph.conf and you removed them live they may reconnect after a restart. Or you put the config into ceph.conf and restart services and see if that helps to recover. Zitat von Tom Verhaeg <t.verhaeg(a)real-websolutions.nl>: > Hi, > > I just installed a new cluster with cephadm > (https://docs.ceph.com/docs/master/cephadm/install/) all was > working fine until I disabled cephx using the following commands: > > ceph config set global auth_cluster_required none > ceph config set global auth_service_required none > ceph config set global auth_client_required none > > Now I keep running into: > > [root@cph1]# ceph health > [errno 13] RADOS permission denied (error connecting to the cluster) > > However, in my /etc/ceph/ceph.conf there is no mention of the > configuration keys mentioned above. Same goes for the cluster folder > in: > > /var/lib/ceph/d5d206f4-8a4f-11ea-9d04-ac1f6bf8b2d3/mgr.cph1.srvnl.nl.qwnssx/config > > How can I remove this setting now? Obviously all ceph * related > commands run into the same RADOS permission denied. > > Thanks, > > Tom > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Benefits of Using CenturyLink Email Services are below
by allewatson0321＠gmail.com 18 Aug '20

18 Aug '20

• Email Size Limit of 20 MB otherwise total storage of 15 GB. • Sending Limit to Contacts up to 100 Contacts at the same time. • Webmail Send Limit of up to 100 / hour. • SMTP Send Limit of up to 500 / hour. • IP Based Send Limit of up to 50 / 5 Minutes. • The user can track the messages sent using the conversation filter • The user can also create various folders and subfolders to organize the mail inbox and then directly can route the similar messages in the designated folder. • The <a href="https://www.emails-login.com/centurylink-email-login/">centurylink.net email login</a> service user can also create a tasks list, to-do lists, and can also set the priority reminder over the important emails. • The user of CenturyLink Email service can also configure the system settings through its own efforts. • The user of <a href="https://www.emails-login.com/centurylink-email-login/">centurylink email login</a> service can also create events like meetings and business appointments.

1 0

Help
by Randy Morgan 17 Aug '20

17 Aug '20

We are seeking information on configuring Ceph to work with Noobaa and NextCloud. Randy -- Randy Morgan CSR Department of Chemistry/BioChemistry Brigham Young University randym(a)chem.byu.edu

3 2

Ceph raw capacity usage does not meet real pool storage usage
by Davood Ghatreh 17 Aug '20

17 Aug '20

Hello everybody, I have a Ceph cluster (14.2.9) running with 18 X 600GB HDD OSDs. There are three pools (size:3, pg_num:64) with an image size of 200GB on each, and there are 6 servers connected to these images via iSCSI and storing about 20 VMs on them. Here is the output of "ceph df": POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 0 B 0 0 B 0 0 B cephfs_metadata 2 17 KiB 22 1.5 MiB 100.00 0 B defaults.rgw.buckets.data 3 0 B 0 0 B 0 0 B defaults.rgw.buckets.index 4 0 B 0 0 B 0 0 B .rgw.root 5 2.0 KiB 5 960 KiB 100.00 0 B default.rgw.control 6 0 B 8 0 B 0 0 B default.rgw.meta 7 393 B 2 384 KiB 100.00 0 B default.rgw.log 8 0 B 207 0 B 0 0 B rbd 9 150 GiB 38.46k 450 GiB 100.00 0 B rbd3 13 270 GiB 69.24k 811 GiB 100.00 0 B rbd2 14 150 GiB 38.52k 451 GiB 100.00 0 B Based on this, I expect about 1.7 TB RAW capacity usage, BUT it is currently about 9TBs! RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 9.8 TiB 870 GiB 9.0 TiB 9.0 TiB 91.35 TOTAL 9.8 TiB 870 GiB 9.0 TiB 9.0 TiB 91.35 And the cluster is down because there is very little capacity remaining. I wonder what makes this and how can I get it fixed. Your help is much appreciated Thank You

1 0

Re: OSD RGW Index 14.2.11 crash
by Dan van der Ster 17 Aug '20

17 Aug '20

Hi, Do you have scsi errors around the time of the crash? `journalctl -k` and look for scsi medium errors. Cheers, Dan On Mon, Aug 17, 2020 at 3:50 PM EDH - Manuel Rios <mriosfer(a)easydatahost.com> wrote: > > Hi , Today one of our SSD dedicated to RGW index crashed, maybe a bug or just osd crashed. > > Our current versión 14.2.11, today we're under heavy object process... aprox 60TB data. > > ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x563f96b550e5] > 2: (()+0x4d72ad) [0x563f96b552ad] > 3: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::v14_2_0::list*, char*)+0xf0e) [0x563f9715aa9e] > 4: (BlueRocksRandomAccessFile::Prefetch(unsigned long, unsigned long)+0x2a) [0x563f9718453a] > 5: (rocksdb::BlockBasedTableIterator<rocksdb::DataBlockIter, rocksdb::Slice>::InitDataBlock()+0x29f) [0x563f9772697f] > 6: (rocksdb::BlockBasedTableIterator<rocksdb::DataBlockIter, rocksdb::Slice>::FindKeyForward()+0x1c0) [0x563f97726bb0] > 7: (()+0x102fd29) [0x563f976add29] > 8: (rocksdb::MergingIterator::Next()+0x42) [0x563f97738162] > 9: (rocksdb::DBIter::Next()+0x1f3) [0x563f97641e53] > 10: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::next()+0x2d) [0x563f975b36bd] > 11: (RocksDBStore::RocksDBTransactionImpl::rm_range_keys(std::string const&, std::string const&, std::string const&)+0x567) [0x563f975beab7] > 12: (BlueStore::_do_omap_clear(BlueStore::TransContext*, std::string const&, unsigned long)+0x72) [0x563f9708f2f2] > 13: (BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>)+0xc16) [0x563f970a6026] > 14: (BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0x5f) [0x563f970a6cbf] > 15: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x13f5) [0x563f970acca5] > 16: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transactio n> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x370) [0x563f970c1100] > 17: (ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TP Handle*)+0x7f) [0x563f96cb6d3f] > 18: (non-virtual thunk to PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x4f) [0x563f96e3015f] > 19: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x4a0) [0x563f96f2a970] > 20: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x298) [0x563f96f32d38] > 21: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x4a) [0x563f96e4486a] > 22: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x5b3) [0x563f96df4c63] > 23: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x362) [0x563f96c34da2] > 24: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x563f96ec37c2] > 25: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x563f96c4fd3f] > 26: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x563f97203c46] > 27: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x563f97206760] > 28: (()+0x7dd5) [0x7f1e504eddd5] > 29: (clone()+0x6d) [0x7f1e4f3ad02d] > > 0> 2020-08-17 15:45:27.609 7f1e2fa82700 -1 *** Caught signal (Aborted) ** > in thread 7f1e2fa82700 thread_name:tp_osd_tp > > ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) > 1: (()+0xf5d0) [0x7f1e504f55d0] > 2: (gsignal()+0x37) [0x7f1e4f2e52c7] > 3: (abort()+0x148) [0x7f1e4f2e69b8] > 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x563f96b55134] > 5: (()+0x4d72ad) [0x563f96b552ad] > 6: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::v14_2_0::list*, char*)+0xf0e) [0x563f9715aa9e] > 7: (BlueRocksRandomAccessFile::Prefetch(unsigned long, unsigned long)+0x2a) [0x563f9718453a] > 8: (rocksdb::BlockBasedTableIterator<rocksdb::DataBlockIter, rocksdb::Slice>::InitDataBlock()+0x29f) [0x563f9772697f] > 9: (rocksdb::BlockBasedTableIterator<rocksdb::DataBlockIter, rocksdb::Slice>::FindKeyForward()+0x1c0) [0x563f97726bb0] > 10: (()+0x102fd29) [0x563f976add29] > 11: (rocksdb::MergingIterator::Next()+0x42) [0x563f97738162] > 12: (rocksdb::DBIter::Next()+0x1f3) [0x563f97641e53] > 13: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::next()+0x2d) [0x563f975b36bd] > 14: (RocksDBStore::RocksDBTransactionImpl::rm_range_keys(std::string const&, std::string const&, std::string const&)+0x567) [0x563f975beab7] > 15: (BlueStore::_do_omap_clear(BlueStore::TransContext*, std::string const&, unsigned long)+0x72) [0x563f9708f2f2] > 16: (BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>)+0xc16) [0x563f970a6026] > 17: (BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0x5f) [0x563f970a6cbf] > 18: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x13f5) [0x563f970acca5] > 19: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transactio n> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x370) [0x563f970c1100] > 20: (ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TP Handle*)+0x7f) [0x563f96cb6d3f] > 21: (non-virtual thunk to PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x4f) [0x563f96e3015f] > 22: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x4a0) [0x563f96f2a970] > 23: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x298) [0x563f96f32d38] > 24: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x4a) [0x563f96e4486a] > 25: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x5b3) [0x563f96df4c63] > 26: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x362) [0x563f96c34da2] > 27: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x563f96ec37c2] > 28: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x563f96c4fd3f] > 29: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x563f97203c46] > 30: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x563f97206760] > 31: (()+0x7dd5) [0x7f1e504eddd5] > 32: (clone()+0x6d) [0x7f1e4f3ad02d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > Any ideas or similar situation? > > > Manuel Ríos Fernández > CEO - Business development > 677677179 · mriosfer(a)easydatahost.com<mailto:mriosfer@easydatahost.com> > No me imprimas si no es necesario. Protejamos el medio ambiente > Este mensaje y, en su caso, los ficheros anexos son confidenciales, especialmente en lo que respecta a los datos personales, y se dirigen exclusivamente al destinatario referenciado. > Si usted no lo es y lo ha recibido por error o tiene conocimiento del mismo por cualquier motivo, le rogamos que nos lo comunique por este medio y proceda a destruirlo o borrarlo, y que en todo caso se abstenga de utilizar, reproducir, alterar, archivar o comunicar a terceros el presente mensaje y ficheros anexos, todo ello bajo pena de incurrir en responsabilidades legales. El emisor no garantiza la integridad, rapidez o seguridad del presente correo, ni se responsabiliza de posibles perjuicios derivados de la captura, incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por terceros. > This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

How to see files in buckets in radosgw object storage in ceph dashboard.?
by sathvik vutukuri 17 Aug '20

17 Aug '20

Hi All, Is there any way to see the list of files under buckets in ceph dashboard for rados object storage. At present I can only see buckets details.

2 1

Looking for Ceph Tech Talks: September 24 and October 22
by Mike Perez 17 Aug '20

17 Aug '20

Hi everyone, We're looking for presentations for the upcoming Ceph Tech Talk dates: * September 24th @ 17:00 UTC * October 22nd @ 17:00 UTC If you're interested or know someone who can present, please let me know! -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

OSD RGW Index 14.2.11 crash
by EDH - Manuel Rios 17 Aug '20

17 Aug '20

Hi , Today one of our SSD dedicated to RGW index crashed, maybe a bug or just osd crashed. Our current versión 14.2.11, today we're under heavy object process... aprox 60TB data. ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x563f96b550e5] 2: (()+0x4d72ad) [0x563f96b552ad] 3: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::v14_2_0::list*, char*)+0xf0e) [0x563f9715aa9e] 4: (BlueRocksRandomAccessFile::Prefetch(unsigned long, unsigned long)+0x2a) [0x563f9718453a] 5: (rocksdb::BlockBasedTableIterator<rocksdb::DataBlockIter, rocksdb::Slice>::InitDataBlock()+0x29f) [0x563f9772697f] 6: (rocksdb::BlockBasedTableIterator<rocksdb::DataBlockIter, rocksdb::Slice>::FindKeyForward()+0x1c0) [0x563f97726bb0] 7: (()+0x102fd29) [0x563f976add29] 8: (rocksdb::MergingIterator::Next()+0x42) [0x563f97738162] 9: (rocksdb::DBIter::Next()+0x1f3) [0x563f97641e53] 10: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::next()+0x2d) [0x563f975b36bd] 11: (RocksDBStore::RocksDBTransactionImpl::rm_range_keys(std::string const&, std::string const&, std::string const&)+0x567) [0x563f975beab7] 12: (BlueStore::_do_omap_clear(BlueStore::TransContext*, std::string const&, unsigned long)+0x72) [0x563f9708f2f2] 13: (BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>)+0xc16) [0x563f970a6026] 14: (BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0x5f) [0x563f970a6cbf] 15: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x13f5) [0x563f970acca5] 16: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transactio n> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x370) [0x563f970c1100] 17: (ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TP Handle*)+0x7f) [0x563f96cb6d3f] 18: (non-virtual thunk to PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x4f) [0x563f96e3015f] 19: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x4a0) [0x563f96f2a970] 20: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x298) [0x563f96f32d38] 21: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x4a) [0x563f96e4486a] 22: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x5b3) [0x563f96df4c63] 23: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x362) [0x563f96c34da2] 24: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x563f96ec37c2] 25: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x563f96c4fd3f] 26: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x563f97203c46] 27: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x563f97206760] 28: (()+0x7dd5) [0x7f1e504eddd5] 29: (clone()+0x6d) [0x7f1e4f3ad02d] 0> 2020-08-17 15:45:27.609 7f1e2fa82700 -1 *** Caught signal (Aborted) ** in thread 7f1e2fa82700 thread_name:tp_osd_tp ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) 1: (()+0xf5d0) [0x7f1e504f55d0] 2: (gsignal()+0x37) [0x7f1e4f2e52c7] 3: (abort()+0x148) [0x7f1e4f2e69b8] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x563f96b55134] 5: (()+0x4d72ad) [0x563f96b552ad] 6: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::v14_2_0::list*, char*)+0xf0e) [0x563f9715aa9e] 7: (BlueRocksRandomAccessFile::Prefetch(unsigned long, unsigned long)+0x2a) [0x563f9718453a] 8: (rocksdb::BlockBasedTableIterator<rocksdb::DataBlockIter, rocksdb::Slice>::InitDataBlock()+0x29f) [0x563f9772697f] 9: (rocksdb::BlockBasedTableIterator<rocksdb::DataBlockIter, rocksdb::Slice>::FindKeyForward()+0x1c0) [0x563f97726bb0] 10: (()+0x102fd29) [0x563f976add29] 11: (rocksdb::MergingIterator::Next()+0x42) [0x563f97738162] 12: (rocksdb::DBIter::Next()+0x1f3) [0x563f97641e53] 13: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::next()+0x2d) [0x563f975b36bd] 14: (RocksDBStore::RocksDBTransactionImpl::rm_range_keys(std::string const&, std::string const&, std::string const&)+0x567) [0x563f975beab7] 15: (BlueStore::_do_omap_clear(BlueStore::TransContext*, std::string const&, unsigned long)+0x72) [0x563f9708f2f2] 16: (BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>)+0xc16) [0x563f970a6026] 17: (BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0x5f) [0x563f970a6cbf] 18: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x13f5) [0x563f970acca5] 19: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transactio n> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x370) [0x563f970c1100] 20: (ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TP Handle*)+0x7f) [0x563f96cb6d3f] 21: (non-virtual thunk to PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x4f) [0x563f96e3015f] 22: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x4a0) [0x563f96f2a970] 23: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x298) [0x563f96f32d38] 24: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x4a) [0x563f96e4486a] 25: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x5b3) [0x563f96df4c63] 26: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x362) [0x563f96c34da2] 27: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x563f96ec37c2] 28: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90f) [0x563f96c4fd3f] 29: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x563f97203c46] 30: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x563f97206760] 31: (()+0x7dd5) [0x7f1e504eddd5] 32: (clone()+0x6d) [0x7f1e4f3ad02d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Any ideas or similar situation? Manuel Ríos Fernández CEO - Business development 677677179 · mriosfer(a)easydatahost.com<mailto:mriosfer@easydatahost.com> No me imprimas si no es necesario. Protejamos el medio ambiente Este mensaje y, en su caso, los ficheros anexos son confidenciales, especialmente en lo que respecta a los datos personales, y se dirigen exclusivamente al destinatario referenciado. Si usted no lo es y lo ha recibido por error o tiene conocimiento del mismo por cualquier motivo, le rogamos que nos lo comunique por este medio y proceda a destruirlo o borrarlo, y que en todo caso se abstenga de utilizar, reproducir, alterar, archivar o comunicar a terceros el presente mensaje y ficheros anexos, todo ello bajo pena de incurrir en responsabilidades legales. El emisor no garantiza la integridad, rapidez o seguridad del presente correo, ni se responsabiliza de posibles perjuicios derivados de la captura, incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por terceros. This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you.

1 0

Ceph OSD Node Maintenance Question
by Matt Dunavant 17 Aug '20

17 Aug '20

Hi all, We just completed maintenance on an OSD node and we ran into an issue where all data seemed to stop flowing while the node was down. We couldn't connect to any of our VMs during that time. I was under the impression that by setting the 'noout' flag, you would not get the rebalance of the data but you would update the pointers to use the 2nd and 3rd copies of the data. Is that not correct, and what is the proper workflow for taking down an OSD node for maintenance? Thanks,

4 5

Ceph Nautilus packages for Ubuntu Focal
by Stefan Kooman 17 Aug '20

17 Aug '20

Hi list, We're wondering if Ceph Nautilus packages will be provided for Ubuntu Focal Fossa (20.04)? You might wonder why one would not just use Ubuntu Bionic (18.04) instead of using the latest LTS. Here is why: a glibc bug in Ubuntu Bionic that *might* affect Open vSwitch (OVS) users [1]. We had quite a few issues with OVS deadlocks on hypervisors, and do not want to risk experiencing the same issues on our Ceph cluster(s). I'm not sure how many of you use OVS for bridging / bonding, but for those who do, running Ceph (Nautlilus / Octopus) on 20.04 would be preferred. Gr. Stefan [1]: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1839592

2 1

osd fast shutdown provokes slow requests
by Manuel Lausch 17 Aug '20

17 Aug '20

Hi, I investigated an other problem with my nautilus 14.2.11 (with 14.2.10 as well) cluster. If I stop the OSDs on one node (systemctl stop ceph-osd.target, or shutdown/reboot) it took mostly several seconds until the cluster detects the OSDs as down and I run in slow requests. I identified the option "osd_fast_shutdown". If I configure it to "false" I see in the ceph.log immediately on shutdown the logs like this: cluster [INF] osd.837 marked itself down If the parameter is true (default setting) The cluster needs some time until I get hundert of lines like this in the ceph.log cluster [DBG] osd.317 reported immediately failed by osd.202 until all detection and peering is done I got slow requests. On smaller clusters, with only 48 OSDs on 4 Nodes the down detection works a lot of faster. But on the big one I need to set this to false to work like expected. I wonder if someone else see this. I think it is OK to short the shutdown process, but it would be nice, if the OSDs tell the mon its shutdown itself Manuel

2 10

Re: radosgw health check url
by Marc Roos 17 Aug '20

17 Aug '20

Yes thanks, that gives at least a 200 OK without xml. However I am still getting these. 7f011d796700 1 ====== req done req=0x562a2a7f85f0 op status=0 http_status=200 latency=0s ====== 7f7c2342a700 1 ====== starting new request req=0x55c6307765f0 ===== 7f7c2342a700 1 ====== req done req=0x55c6307765f0 op status=0 http_status=200 latency=0s ====== 7f0192499700 0 WARNING: RGWRados::log_usage(): user name empty (bucket=), skipping 7f7cc117f700 0 WARNING: RGWRados::log_usage(): user name empty (bucket=), skipping 7f0119f8f700 1 ====== starting new request req=0x562a2a8105f0 ===== -----Original Message----- To: Janne Johansson Cc: Marc Roos; ceph-users Subject: Re: [ceph-users] Re: radosgw health check url Have you tried "/swift/healthcheck"? Thanks! ––––––––– Sébastien Han Senior Principal Software Engineer, Storage Architect "Always give 100%. Unless you're giving blood." On Mon, Aug 17, 2020 at 8:58 AM Janne Johansson <icepic.dz(a)gmail.com> wrote: > > Den sön 16 aug. 2020 kl 21:18 skrev Marc Roos <M.Roos(a)f1-outsourcing.eu>: > > > I have just put radosgw behind haproxy, and wondered if there is a > > health check endpoint. If not maybe it should be > > > Yes, very good suggestion, both for having something where the rgw > could actually state if it "feels mostly ok" in some decent fashion, > and then of course so rgw can log something nice like "served health > check data, 200 - ok" instead of "stupid broken connection attempt failed". > > -- > May the most significant bit of your life be positive. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io

1 0

Re: radosgw health check url
by Janne Johansson 17 Aug '20

17 Aug '20

Den sön 16 aug. 2020 kl 21:18 skrev Marc Roos <M.Roos(a)f1-outsourcing.eu>: > I have just put radosgw behind haproxy, and wondered if there is a > health check endpoint. If not maybe it should be Yes, very good suggestion, both for having something where the rgw could actually state if it "feels mostly ok" in some decent fashion, and then of course so rgw can log something nice like "served health check data, 200 - ok" instead of "stupid broken connection attempt failed". -- May the most significant bit of your life be positive.

1 0

Large omap
by Szabo, Istvan (Agoda) 17 Aug '20

17 Aug '20

Hi, I’m suffering with our large omap object in the cluster since my colleague updated from jewel to luminous 12.2.8. This warn is still here more than a year ago. LARGE_OMAP_OBJECTS 1 large omap objects 1 large objects found in pool 'default.rgw.log' Search the cluster log for 'Large omap object found' for more details. Based on the articles that I’ve read it is due to the usage logs. On the weekend I’ve set the rgw_enable_usage_log = false and restarted all the 3 rgw services. I trim the logs as some people on the internet suggested, but it is super slow. I trim the logs month by month like this: radosgw-admin usage trim --end-date=2019-09-01 --yes-i-really-mean-it This command took yesterday more than a day to finish :/ I hope at least it is not increasing anymore since I turned it to false. The luminous default thresholds are: Size threshold: 1gb Key threshold: 2000000 Mine is: 7gb and 41246592 2020-08-16 09:21:11.549128 osd.45 osd.45 10.118.191.247:6824/2262114 2 : cluster [WRN] Large omap object found. Object: 4:b172cd59:usage::usage.26:head Key count: 41246592 Size (bytes): 7736069974 Based on this command here is my large omap: for i in `ceph pg ls-by-pool default.rgw.log | tail -n +2 | awk '{print $1}'`; do echo -n "$i: "; ceph pg $i query |grep num_large_omap_objects | head -1 | awk '{print $2}'; done | grep ": 1" 4.d: 1 I’ve initiated a deep scrub after it and it is still running after 12 hours and make stuck pgs: health: HEALTH_ERR 3 stuck requests are blocked > 4096 sec. Implicated osds 45 It stuck for couple of secs, and after go back to normal, but this is continously. At the moment no impact yet, but I think it will have soon. What should I do? What is the best and easiest way to fix my issue? We have serious impact with the moving of this pg when an osd crash because I guess it is big and not healthy. Here are the omap list: for i in `rados -p default.rgw.log ls`; do echo -n “$i:“; rados -p default.rgw.log listomapkeys $i |wc -l; done > /tmp/omapkeys cat /tmp/omapkeys “obj_delete_at_hint.0000000078:“0 “meta.history:“0 “obj_delete_at_hint.0000000070:“0 “obj_delete_at_hint.0000000104:“0 “obj_delete_at_hint.0000000026:“0 “obj_delete_at_hint.0000000028:“0 “obj_delete_at_hint.0000000040:“0 “obj_delete_at_hint.0000000015:“0 “obj_delete_at_hint.0000000069:“0 “obj_delete_at_hint.0000000095:“0 “obj_delete_at_hint.0000000003:“0 “obj_delete_at_hint.0000000047:“0 “obj_delete_at_hint.0000000035:“0 “obj_delete_at_hint.0000000037:“0 “obj_delete_at_hint.0000000024:“0 “obj_delete_at_hint.0000000000:“0 “obj_delete_at_hint.0000000031:“0 “obj_delete_at_hint.0000000076:“0 “obj_delete_at_hint.0000000113:“0 “obj_delete_at_hint.0000000005:“0 “obj_delete_at_hint.0000000011:“0 “obj_delete_at_hint.0000000058:“0 “obj_delete_at_hint.0000000052:“0 “obj_delete_at_hint.0000000088:“0 “obj_delete_at_hint.0000000080:“0 “obj_delete_at_hint.0000000090:“0 “obj_delete_at_hint.0000000110:“0 “obj_delete_at_hint.0000000096:“0 “obj_delete_at_hint.0000000087:“0 “obj_delete_at_hint.0000000008:“0 “obj_delete_at_hint.0000000006:“0 “obj_delete_at_hint.0000000029:“0 “obj_delete_at_hint.0000000089:“0 “obj_delete_at_hint.0000000022:“0 “bilog.trim:“0 “obj_delete_at_hint.0000000016:“0 “obj_delete_at_hint.0000000041:“0 “obj_delete_at_hint.0000000018:“0 “obj_delete_at_hint.0000000092:“0 “obj_delete_at_hint.0000000014:“0 “obj_delete_at_hint.0000000112:“0 “obj_delete_at_hint.0000000007:“0 “obj_delete_at_hint.0000000021:“0 “obj_delete_at_hint.0000000064:“0 “obj_delete_at_hint.0000000071:“0 “obj_delete_at_hint.0000000074:“0 “obj_delete_at_hint.0000000081:“0 “obj_delete_at_hint.0000000009:“0 “obj_delete_at_hint.0000000121:“0 “obj_delete_at_hint.0000000125:“0 “obj_delete_at_hint.0000000082:“0 “obj_delete_at_hint.0000000105:“0 “obj_delete_at_hint.0000000059:“0 “obj_delete_at_hint.0000000077:“0 “obj_delete_at_hint.0000000032:“0 “obj_delete_at_hint.0000000053:“0 “obj_delete_at_hint.0000000091:“0 “obj_delete_at_hint.0000000065:“0 “obj_delete_at_hint.0000000083:“0 “obj_delete_at_hint.0000000010:“0 “obj_delete_at_hint.0000000045:“0 “obj_delete_at_hint.0000000002:“0 “obj_delete_at_hint.0000000116:“0 “obj_delete_at_hint.0000000034:“0 “obj_delete_at_hint.0000000101:“0 “obj_delete_at_hint.0000000079:“0 “obj_delete_at_hint.0000000049:“0 “obj_delete_at_hint.0000000117:“0 “obj_delete_at_hint.0000000044:“0 “obj_delete_at_hint.0000000066:“0 “obj_delete_at_hint.0000000068:“0 “obj_delete_at_hint.0000000085:“0 “obj_delete_at_hint.0000000073:“0 “obj_delete_at_hint.0000000038:“0 “obj_delete_at_hint.0000000118:“0 “obj_delete_at_hint.0000000036:“0 “obj_delete_at_hint.0000000103:“0 “obj_delete_at_hint.0000000119:“0 “obj_delete_at_hint.0000000098:“0 “obj_delete_at_hint.0000000027:“0 “obj_delete_at_hint.0000000019:“0 “obj_delete_at_hint.0000000039:“0 “obj_delete_at_hint.0000000100:“0 “obj_delete_at_hint.0000000093:“0 “obj_delete_at_hint.0000000004:“0 “obj_delete_at_hint.0000000063:“0 “obj_delete_at_hint.0000000122:“0 “obj_delete_at_hint.0000000057:“0 “obj_delete_at_hint.0000000054:“0 “obj_delete_at_hint.0000000114:“0 “data_log.0:“0 “obj_delete_at_hint.0000000012:“0 “obj_delete_at_hint.0000000084:“0 “obj_delete_at_hint.0000000043:“0 “obj_delete_at_hint.0000000111:“0 “obj_delete_at_hint.0000000048:“0 “obj_delete_at_hint.0000000020:“0 “obj_delete_at_hint.0000000099:“0 “obj_delete_at_hint.0000000056:“0 “obj_delete_at_hint.0000000072:“0 “obj_delete_at_hint.0000000062:“0 “obj_delete_at_hint.0000000115:“0 “obj_delete_at_hint.0000000033:“0 “obj_delete_at_hint.0000000025:“0 “obj_delete_at_hint.0000000123:“0 “obj_delete_at_hint.0000000108:“0 “obj_delete_at_hint.0000000094:“0 “obj_delete_at_hint.0000000060:“0 “obj_delete_at_hint.0000000109:“0 “obj_delete_at_hint.0000000013:“0 “obj_delete_at_hint.0000000051:“0 “obj_delete_at_hint.0000000106:“0 “obj_delete_at_hint.0000000017:“0 “obj_delete_at_hint.0000000023:“0 “obj_delete_at_hint.0000000107:“0 “obj_delete_at_hint.0000000046:“0 “obj_delete_at_hint.0000000067:“0 “obj_delete_at_hint.0000000086:“0 “obj_delete_at_hint.0000000050:“0 “obj_delete_at_hint.0000000102:“0 “obj_delete_at_hint.0000000124:“0 “obj_delete_at_hint.0000000001:“0 “obj_delete_at_hint.0000000126:“0 “obj_delete_at_hint.0000000055:“0 “obj_delete_at_hint.0000000061:“0 “obj_delete_at_hint.0000000097:“0 “obj_delete_at_hint.0000000042:“0 “obj_delete_at_hint.0000000075:“0 “obj_delete_at_hint.0000000120:“0 “obj_delete_at_hint.0000000030:“0 Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

radosgw (ceph ) time logging
by Marc Roos 16 Aug '20

16 Aug '20

I do not need ceph/radosgw to add the time to logs going to syslog. How to turn this off, or where to change the logging format? Aug 16 21:15:24 c03 radosgw: 2020-08-16 21:15:24.657 7f41754e0700 1 ====== starting new request req=0x55cdb3dfe5f0 ===== Aug 16 21:15:24 c03 radosgw: 2020-08-16 21:15:24.659 7f41754e0700 1 ====== req done req=0x55cdb3dfe5f0 op status=0 http_status=200 latency=0.00199996s ====== Aug 16 21:15:24 c03 radosgw: 2020-08-16 21:15:24.727 7f417bced700 1 ====== starting new request req=0x55cdb2b485f0 ===== Aug 16 21:15:24 c03 radosgw: 2020-08-16 21:15:24.729 7f417bced700 1 ====== req done req=0x55cdb2b485f0 op status=0 http_status=200 latency=0.00199996s ====== Aug 16 21:15:25 c04 radosgw: 2020-08-16 21:15:25.987 7f83d20c3700 1 ====== starting new request req=0x561ed8a625f0 ===== Aug 16 21:15:25 c04 radosgw: 2020-08-16 21:15:25.989 7f83d20c3700 1 ====== req done req=0x561ed8a625f0 op status=0 http_status=200 latency=0.00199992s ====== Aug 16 21:15:33 c04 radosgw: 2020-08-16 21:15:33.620 7f83cd0b9700 1 ====== starting new request req=0x561ed8a625f0 ===== Aug 16 21:15:33 c04 radosgw: 2020-08-16 21:15:33.622 7f83cd0b9700 1 ====== req done req=0x561ed8a625f0 op status=0 http_status=200 latency=0.00199992s ====== Aug 16 21:15:44 c03 radosgw: 2020-08-16 21:15:44.663 7f4168cc7700 1 ====== starting new request req=0x55cdb19e45f0 ===== Aug 16 21:15:44 c03 radosgw: 2020-08-16 21:15:44.665 7f4168cc7700 0 ERROR: client_io->complete_request() returned Connection reset by peer Aug 16 21:15:44 c03 radosgw: 2020-08-16 21:15:44.665 7f4168cc7700 1 ====== req done req=0x55cdb19e45f0 op status=0 http_status=200 latency=0.00199996s ======

1 0

radosgw health check url
by Marc Roos 16 Aug '20

16 Aug '20

I have just put radosgw behind haproxy, and wondered if there is a health check endpoint. If not maybe it should be, now I am getting this in the logs. Aug 16 21:15:24 c03 radosgw: 2020-08-16 21:15:24.657 7f41754e0700 1 ====== starting new request req=0x55cdb3dfe5f0 ===== Aug 16 21:15:24 c03 radosgw: 2020-08-16 21:15:24.659 7f41754e0700 1 ====== req done req=0x55cdb3dfe5f0 op status=0 http_status=200 latency=0.00199996s ====== Aug 16 21:15:24 c03 radosgw: 2020-08-16 21:15:24.727 7f417bced700 1 ====== starting new request req=0x55cdb2b485f0 ===== Aug 16 21:15:24 c03 radosgw: 2020-08-16 21:15:24.729 7f417bced700 1 ====== req done req=0x55cdb2b485f0 op status=0 http_status=200 latency=0.00199996s ====== Aug 16 21:15:25 c04 radosgw: 2020-08-16 21:15:25.987 7f83d20c3700 1 ====== starting new request req=0x561ed8a625f0 ===== Aug 16 21:15:25 c04 radosgw: 2020-08-16 21:15:25.989 7f83d20c3700 1 ====== req done req=0x561ed8a625f0 op status=0 http_status=200 latency=0.00199992s ====== Aug 16 21:15:33 c04 radosgw: 2020-08-16 21:15:33.620 7f83cd0b9700 1 ====== starting new request req=0x561ed8a625f0 ===== Aug 16 21:15:33 c04 radosgw: 2020-08-16 21:15:33.622 7f83cd0b9700 1 ====== req done req=0x561ed8a625f0 op status=0 http_status=200 latency=0.00199992s ====== Aug 16 21:15:44 c03 radosgw: 2020-08-16 21:15:44.663 7f4168cc7700 1 ====== starting new request req=0x55cdb19e45f0 ===== Aug 16 21:15:44 c03 radosgw: 2020-08-16 21:15:44.665 7f4168cc7700 0 ERROR: client_io->complete_request() returned Connection reset by peer Aug 16 21:15:44 c03 radosgw: 2020-08-16 21:15:44.665 7f4168cc7700 1 ====== req done req=0x55cdb19e45f0 op status=0 http_status=200 latency=0.00199996s ======

1 0

Can't add OSD id in manual deploy
by Joshua Schaeffer 15 Aug '20

15 Aug '20

Hey all, I'm trying to deploy Ceph 15.2.4 on Ubuntu 20.04 and I am going through the manual deploy process [1]. I was able to successfully bootstrap the monitor and manager and am now trying to add the OSD's, but the `ceph-volume` command is hanging when running `ceph osd new`. It appears the command hangs for about 30 minutes and then times out with this error: user@node1:~$ sudo ceph-volume lvm create --data /dev/sdc Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9a305232-6b0a-4fde-9685-68fe6150da0e stderr: [errno 110] RADOS timed out (error connecting to the cluster) --> RuntimeError: Unable to create a new OSD id I have also tried to manually run the `ceph osd new` command, but it results in the same error. My cluster shows as up (at least the monitor and manager): user@node1:~$ sudo ceph -s cluster: id: 6aacdfe1-42e4-4106-9503-3e16e9812195 health: HEALTH_WARN 1 monitors have not enabled msgr2 Reduced data availability: 1 pg inactive OSD count 0 < osd_pool_default_size 3 services: mon: 1 daemons, quorum bllcloudceph01 (age 23h) mgr: bllcloudceph01(active, since 23h) osd: 0 osds: 0 up, 0 in data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 1 unknown And ceph appears to be listening: user@node1:~$ sudo ss -tupanl | grep ceph tcp LISTEN 0 512 10.2.22.1:6789 0.0.0.0:* users:(("ceph-mon",pid=136244,fd=25)) tcp LISTEN 0 512 10.2.22.1:6800 0.0.0.0:* users:(("ceph-mgr",pid=138125,fd=25)) tcp LISTEN 0 512 10.2.22.1:6801 0.0.0.0:* users:(("ceph-mgr",pid=138125,fd=26)) It seems like I'm missing something simple, but I can't spot it. Can anybody point me in the right direction? [1] https://docs.ceph.com/docs/master/install/manual-deployment/ -- Thanks, Joshua Schaeffer

2 6

Error adding host in ceph-iscsi
by Gesiel Galvão Bernardes 15 Aug '20

15 Aug '20

Hi, I tryng add a VMware host in Ceph ISCSI. I followed exactly as is in guide. But when add the iscsi gateway IP in "Dynamic Discovery", and "rescan adapter", not loaded the "Paths". In vmkernel.log, I receive this messages: 2020-08-15T13:50:36.166Z cpu21:2103927)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1235: vmhba64:CH:0 T:0 CN:0: Connection rx notifying failure: Fai led to Receive. State=Bound 2020-08-15T13:50:36.166Z cpu21:2103927)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1236: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-08-15T13:50:36.166Z cpu21:2103927)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1237: Conn [CID: 0 L: 192.168.201.109:59665 R: 192.168.201.1:3260 ] 2020-08-15T13:50:36.167Z cpu21:2103927)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:517: vmhba64:CH:1 T:0 CN:0: Failed to receive data: Connec tion closed by peer 2020-08-15T13:50:36.167Z cpu21:2103927)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:525: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-08-15T13:50:36.167Z cpu21:2103927)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:526: Conn [CID: 0 L: 192.168.201.109:22890 R: 192.168.201. 2:3260] 2020-08-15T13:50:36.167Z cpu21:2103927)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1235: vmhba64:CH:1 T:0 CN:0: Connection rx notifying failure: Fai led to Receive. State=Bound 2020-08-15T13:50:36.167Z cpu21:2103927)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1236: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-08-15T13:50:36.167Z cpu21:2103927)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1237: Conn [CID: 0 L: 192.168.201.109:22890 R: 192.168.201.2:3260 ] 2020-08-15T13:50:36.419Z cpu32:2104379)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba64:CH:0 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:2) 2020-08-15T13:50:36.419Z cpu32:2104379)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-08-15T13:50:36.419Z cpu32:2104379)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 192.168.201.109:59665 R: 192.168.201.1:3 260] 2020-08-15T13:50:36.424Z cpu32:2104379)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba64:CH:1 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:2) 2020-08-15T13:50:36.424Z cpu32:2104379)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-08-15T13:50:36.424Z cpu32:2104379)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 192.168.201.109:22890 R: 192.168.201.2:3 260] Can anyone help-me? Regards, Gesiel

1 0

How big mon osd down out interval could be?
by huxiaoyu＠horebdata.cn 15 Aug '20

15 Aug '20

Dear Ceph folks, Suppose i have extremely reliable OSDs, which almost never fails (of course an imaginary ideal case). The OSDs may still go UPs or DOWNs due to network issues, but i would like to make OSD never OUT and thus no need for backfilling for OSD failure. To achieve the above goal, i can change mon osd down out interval from 300s to a much large value. My question is, what is the side effect of doing so, and what is the upper limit for mon osd down out interval ? best regards, samuel huxiaoyu(a)horebdata.cn

1 0

Re: how to handle incomplete PGs
by Eugen Block 15 Aug '20

15 Aug '20

Please provide more details about your environment, otherwise it's just guessing what could have happened. Zitat von huxiaoyu(a)horebdata.cn: > I am using replicated pool, and min_size=1. I do not have any disk > failure, so i do not expect incomplete PGs, but it appeared after > OSD flaped. > > > > huxiaoyu(a)horebdata.cn > > From: Eugen Block > Date: 2020-08-15 09:39 > To: huxiaoyu > CC: ceph-users > Subject: Re: [ceph-users] how to handle incomplete PGs > Hi, > > did you wait for the backfill to complete before removing the old > drives? What is your environment? Are the affected PGs from an EC > pool? Does [1] apply to you? > > [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-July/035743.html > > > Zitat von huxiaoyu(a)horebdata.cn: > >> Dear Ceph folks, >> >> Recently i encountered incomplete PGs when replacing an OSD node >> with new handware. I noticed multiple OSD ups and downs, and >> eventually a few PGs got stucked at PG incomplete status. >> >> Questions 1: is there a reliable way to avoid the occurence of >> incomplete PGs? >> 2: is there a good tool or scriptes to handle >> incomplete PGs without lossing data >> >> best regards, >> >> samuel >> >> >> >> huxiaoyu(a)horebdata.cn >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: how to handle incomplete PGs
by Eugen Block 15 Aug '20

15 Aug '20

Hi, did you wait for the backfill to complete before removing the old drives? What is your environment? Are the affected PGs from an EC pool? Does [1] apply to you? [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-July/035743.html Zitat von huxiaoyu(a)horebdata.cn: > Dear Ceph folks, > > Recently i encountered incomplete PGs when replacing an OSD node > with new handware. I noticed multiple OSD ups and downs, and > eventually a few PGs got stucked at PG incomplete status. > > Questions 1: is there a reliable way to avoid the occurence of > incomplete PGs? > 2: is there a good tool or scriptes to handle > incomplete PGs without lossing data > > best regards, > > samuel > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

how to handle incomplete PGs
by huxiaoyu＠horebdata.cn 15 Aug '20

15 Aug '20

Dear Ceph folks, Recently i encountered incomplete PGs when replacing an OSD node with new handware. I noticed multiple OSD ups and downs, and eventually a few PGs got stucked at PG incomplete status. Questions 1: is there a reliable way to avoid the occurence of incomplete PGs? 2: is there a good tool or scriptes to handle incomplete PGs without lossing data best regards, samuel huxiaoyu(a)horebdata.cn

1 0

CephFS clients waiting for lock when one of them goes slow
by Petr Belyaev 15 Aug '20

15 Aug '20

Hi all, I have a very small Ceph setup - 3 OSDs, 3 MDS, 3 MON, CephFS with ~20 ceph-fuse clients connected. All run version 14.2.9 (both servers and clients). Some clients work with a relatively large CephFS directory, few tens of thousands of files, with lots of files being added and deleted all the time. Almost every day I see exactly the same pattern when system goes under load: - One of the clients starts running slow (e.g. high CPU or out-of-memory, sometimes with OOM killer intervention) - A number of requests from other clients gets blocked for ~15 seconds waiting for the (r/w/x)lock on other files in the same directory - Objecter requests empty - No requests visibly running in the dump_ops_in_flight output, only those waiting for the lock MDS cache size is 1.5 gb, total capacity - around 1TB, usual load is ~300 [rw]ops, 3-4 mb/s, most of the config is the default one. Have somebody seen similar issues before? Best regards, Petr Belyaev

4 3

Nautilus slow using "ceph tell osd.* bench"
by Jim Forde 14 Aug '20

14 Aug '20

I have 2 clusters. Cluster 1 started at Hammer and has upgraded through the versions all the way to Nautilus 14.2.10 (Luminous to Nautilus in July 2020) . Cluster 2 started as Luminous and is now Nautilus 14.2.2 (Upgraded in September 2019) The clusters are basically identical 5 OSD Nodes with 6 osd's per node. They are both using disk drives. No SSD's. Prior to upgrading Cluster 1 running "ceph tell osd.0 bench -f plain" produced similar results across both clusters. ceph tell osd.0 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 0.954819 sec at 1.0 GiB/sec 268 IOPS Now cluster 1 results are terrible, about 25% from before the upgrade. ceph tell osd.0 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 4.03434 sec at 254 MiB/sec 63 IOPS Ceph -s shows HEALTH_OK. Dashboard looks good. 2 pools MON Dump min_mon_release 14 (nautilus) OSD Dump require_min_compat_client luminous min_compat_client jewel require_osd_release nautilus Not sure what is causing the slow performance. Ideas?

3 6

Resolving a pg inconsistent Issue
by Steven Pine 14 Aug '20

14 Aug '20

Hi Ceph users, We are running a suse SES 5.5 cluster that's largely based on luminous with some mimic backports. We've been doing some large reshuffling from adding in additional OSDs and during this process we have an inconsistent pg group, investigation suggests there was a read error. We would like to target it with a deep-scrub and possibly repair attempt before marking the individual osd down and replacing the drive, but because of the weeks of reshuffling, our cluster is behind on deep-scrubs and it seemingly is refusing to scrub the marked pg, or at least it is scheduling other pg for deep-scrubs first. Is there any method we can use to grant deep-scrub priority? How about adjusting deep-scrub timeframes to be 6 months temporarily, would that allow us to force the desired pg deep-scrub to occur prior? Or could set some of the no scrub flags and proceed with a pg repair attempt? Thank you for any suggestions or advice, -- Steven Pine webair.com *P* 516.938.4100 x *E * steven.pine(a)webair.com <https://www.facebook.com/WebairInc/> <https://www.linkedin.com/company/webair>

2 1

Radosgw Multiside Sync
by Ansgar Jazdzewski 14 Aug '20

14 Aug '20

Hi Folks, i'am trying to move from our own custom bucket synchronization to the rados-gateway build in one. Multisite setup is working https://docs.ceph.com/docs/master/radosgw/multisite/ All buckes and users are visible in both clusters Next i tried to setup the multi-side-sync https://docs.ceph.com/docs/master/radosgw/multisite-sync-policy/ for the first test i liked to have a full symmetric setup so i configured it as followed radosgw-admin sync group create \ --group-id=group1 \ --status=allowed radosgw-admin sync group flow create \ --group-id=group1 \ --flow-id=flow-mirror \ --flow-type=symmetrical \ --zones=* radosgw-admin sync group pipe create \ --group-id=group1 \ --pipe-id=pipe1 \ --source-zones='*' \ --source-bucket='*' \ --dest-zones='*' \ --dest-bucket='*' radosgw-admin sync group modify \ --group-id=group1 \ --status=enabled radosgw-admin period update \ --commit but the objects are not copied around do i need to start the copy-process? can i find some debug information somewhere? Thanks, Ansgar

2 4

Re: SED drives ,*how to fio test all disks, poor performance
by Ed Kalk 14 Aug '20

14 Aug '20

screenshot attached showing the IOPs and LAtency from iostat -xtc 2 On 8/14/2020 9:09 AM, Ed Kalk wrote: > ubuntu@ubuntu:/mnt$ sudo fio > --filename=/mnt/sda1/file1.fio:/mnt/sdb1/file2.fio:/mnt/sdc1/file3.fio:/mnt/sdd1/file4.fio:/mnt/sde1/file5.fio:/mnt/sdf1/file6.fio:/mnt/sdg1/file7.fio:/mnt/sdh1/file8.fio:/mnt/sdi1/file9.fio:/mnt/sdj1/file10.fio:/mnt/sdk1/file11.fio:/mnt/sdl1/file12.fio > --size=2GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio > --iodepth=256 --runtime=500 --numjobs=12 --time_based > --group_reporting --name=iops-test-job > > ^generate fio to all disks > > ----- > # iostat -xtc 2 > > ^monitor IOPs to all disks > ----- > > ^how to test “fio” to multiple disks and monitor via “iostat -xtc 2“. > I booted via live CD, made test partitions, file systems and mounted. > disks took a while to initialize and settle in after filesystem > create. monitored via iostat till all disks were quiet. then ran fio > to test. > > we observed the fist 6 disks (seagate constellation v1 SED) have a > problem, but next 6 don’t. (seagate constellation v2 non-SED) > > The “w_await” shows latency is 100x on v1’s, this was a reproduction > of how they performed in ceph. > > ***suspecting disk firmware, will try updating them. > > -Edward Kalk > > On Aug 9, 2020, at 10:44 AM, Edward kalk <ekalk(a)socket.net> wrote: > > Thank you for the reply Anthony. > Yes the R510 is of a previous gen. The H700 isn’t my top suspect > though. It’s been clocked to pull 15,000+ iops (with SSD). I’m > familiar with bottlenecking an hba with 24 disk chassis. I only have 6 > ceph OSDs on the H700. (this system should be overkill for that. > (ejected all other disks, only 8 inserted) > I’ll try investigating for FW updates on disks and hba. > will also take another look at disk health via smart. they are > repurposed disks and have seen action to say the least. thanks. > -Ed > >>>>> > >>>>>> On Aug 8, 2020, at 6:56 PM, Anthony D'Atri > <anthony.datri(a)gmail.com> wrote: > >>>>>> > >>>>>> Ugh, that looks like a pretty old HBA (and system). > >>>>>> > >>>>>> On an R730xd I found an H330 mini to be something of a > bottleneck with 24x SATA SSD. > >>>>>> > >>>>>> o Check your drives for available firmware updates > >>>>>> > >>>>>> o Check your HBA for a firmware update > >>>>>> > >>>>>> o Check that your HBA’s firmware is supported with your > kernel’s driver version — there should be a matrix at Dell somewhere > >>>>>> > >>>>>> o If you have a BBU/supercap, check its status (perccli /c0 … ) > >>>>>> > >>>>>> o Check dmesg/logs for HBA errors > >>>>>> > >>>>>> o Check smartctl -a output on the drives, look for wear, > reallocated sectors, errors > >>>>>> > >>>>>>>> On Aug 8, 2020, at 2:55 PM, Edward kalk <ekalk(a)socket.net> wrote: > >>>>>>> R0, the controller config. (faux-JBOD) > >>>>>>> Single disk group. NoN-Raid. > >>>>>>> > >>>>>>> -Ed > >>>>>>> > >>>>>>>>> On Aug 8, 2020, at 9:20 AM, Anthony D'Atri > <anthony.datri(a)gmail.com> wrote: > >>>>>>>> By R0 you mean hdparm ? > >>>>>>>> > >>>>>>>>> On Aug 8, 2020, at 3:49 AM, Edward kalk <ekalk(a)socket.net> > wrote: > >>>>>>>>> > >>>>>>>>> Im getting poor performance with 5 of my OSDs, Seagate > Constellation ES SED (1) 10k SAS 2TB 3.5 drives. > >>>>>>>>> disk write latency keeps drifting high , > >>>>>>>>> 100ms-230ms on writes. the other 30 OSDs are performing > well. avg latency 10-20ms > >>>>>>>>> > >>>>>>>>> We observe stats via “iostat -xtc 2” on CEPH server. w_await > showing writes to disk latency. > >>>>>>>>> > >>>>>>>>> **anyone else have poor latency on SEDs? > >>>>>>>>> > >>>>>>>>> The disks were included on a server we acquired. Didn’t know > they would be SEDs. > >>>>>>>>> I’ve read that seagate SED vs. non SED shouldn’t perform > differently, but looks like they are awful. All the SEDs are in the > newly added CEPH node. A Dell R510 with H700 raid card. all disks in > entire cluster are R0, direct write to disk, no cache. > >>>>>>>>> The new server and disk control spec out at about 10x the > ability and IOPs of other servers, so I suspect the SEDs. > >>>>>>>>> > >>>>>>>>> -Ed > >>>>>>>>> _______________________________________________ > >>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io > >>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>> -- > >>> Thank you for your time, > >>> > >>> > >>> > >>> Edward H. Kalk IV > >>> Information Technology Dept. > >>> Server Specialist > >>> Datacenter Virtualization and Storage Systems > >>> Socket Telecom, LLC. > >>> 2703 Clark Lane > >>> Columbia, MO 65202 > >>> 573-817-0000 or 800-socket3 X218 > >> -- > > -- Thank you for your time, Edward H. Kalk IV Information Technology Dept. Server Specialist Datacenter Virtualization and Storage Systems Socket Telecom, LLC. 2703 Clark Lane Columbia, MO 65202 573-817-0000 or 800-socket3 X218

1 0

How to separate WAL DB and DATA using cephadm or other method?
by Popoi Zen 14 Aug '20

14 Aug '20

Hi, I just want to know if it possible to separate my NVME disk and HDD disk roles on cephadm. Example I have an NVME paritition: /dev/nvme0n0 and HDD: /dev/sdb. How to pair that 2 disk, so I can use /dev/sdb as OSD and the whole /dev/nvme0n0 as WAL DB?

1 0

RGW Lifecycle Processing and Promote Master Process
by Alex Hussein-Kershaw 14 Aug '20

14 Aug '20

Hi, I've previously discussed some issues I've had with the RGW lifecycle processing. I've discovered that the root cause of my problem is that: * I'm running a multisite configuration * Life cycle processing is done on the master site each night. `radosgw-admin lc list` correctly returns all buckets with lc config. * I simulate the master site being destroyed from my VM host. * I promote the secondary site to master following the instructions here: https://docs.ceph.com/docs/master/radosgw/multisite/ * The new master site isn't doing any lifecycle processing. `radosgw-admin lc list` returns empty. * I recreate a cluster and pair it with the new master site to get back to having multisite redundancy. * Neither site is doing any lifecycle processing. `radosgw-admin lc list` returns empty. So in the process of failover/recovery I have gone from having two paired clusters performing lifecycle processing, to two paired clusters NOT performing lifecycle processing. Is this behaviour expected? I've found `radosgw-admin lc reshard fix` will "remind" the cluster that I run it on that it needs to do lifecycle processing. Although I found no mention of having to use this in the docs, for that command the docs state it's only relevant on earlier Ceph versions. I'm running Nautilus 14.2.9. In addition, if I have two healthy clusters paired in a multisite system, and swap the master cluster by promoting the non-master, the demoted cluster seems to still continue doing lifecycle processing, while the promote does not. If I run `radosgw-admin lc reshard fix` on the promoted cluster, then both clusters seem to claim they are doing the processing. Is this a happy state to be in? Does anyone have any experience with this? Thanks, Alex

1 0

Single node all-in-one install for testing
by Richard W.M. Jones 14 Aug '20

14 Aug '20

I have one spare machine with a single 1TB on it, and I'd like to test a local Ceph install. This is just for testing, I don't care that it won't have redundancy, failover, etc. Is there any canonical documentation for this case? - - - Longer story is this morning I found this documentation through many Google searches: https://medium.com/@balderscape/setting-up-a-virtual-single-node-ceph-stora… It kind of works up to a point. I was able to get Ceph installed, get both the command line and web interfaces up and running, and got as far as: # ceph status cluster: id: 723e09aa-dbd3-11ea-8587-94c691189836 health: HEALTH_WARN Reduced data availability: 1 pg inactive OSD count 0 < osd_pool_default_size 3 services: mon: 1 daemons, quorum dev1 (age 77m) mgr: dev1.ziibhq(active, since 76m) osd: 0 osds: 0 up, 0 in data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 1 unknown # ceph orch device ls HOST PATH TYPE SIZE DEVICE AVAIL REJECT REASONS dev1 /dev/sdb hdd 14.9G Cruzer_Blade_4C530210050318117591 True dev1 /dev/sda hdd 931G WDC_WD10JFCX-68N_WD-WXD1AB73AUX5 False LVM detected, locked (The USB stick is left over from the install and isn't suitable as an OSD). However I've been completely unable to add any OSDs using the local disk. The disk (/dev/sda3) is a PV and there is plenty of space. I even created some devices that I was hoping to use as OSDs: # lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert osd1 rhel_dev1 -wi-a----- 100.00g osd2 rhel_dev1 -wi-a----- 100.00g osd3 rhel_dev1 -wi-a----- 100.00g root rhel_dev1 -wi-ao---- 50.00g swap rhel_dev1 -wi-ao---- <7.84g but it simply doesn't do anything when I try to add them: # ceph orch daemon add osd localhost:rhel_dev1/osd1 (no error, exit code 0) So I guess the fact that the device is marked as "AVAIL = False" and "locked" is bad somehow. Can I add the OSDs anyway? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html

2 1

Ceph Tech Talk: A Different Scale – Running small ceph clusters in multiple data centers by Yuval Freund
by Mike Perez 13 Aug '20

13 Aug '20

Hi everyone, Get ready for July 23rd at 17:00 UTC another Ceph Tech Talk but a different scale: running small Ceph clusters in multiple data centers. Thanks to Yuval Freund for giving time to provide us content this month. You can find the calendar invite and archive here: https://ceph.io/ceph-tech-talks/ -- Mike Perez He/Him Ceph Community Manager Red Hat Los Angeles <https://www.redhat.com> thingee(a)redhat.com <mailto:thingee@redhat.com> M: 1-951-572-2633 <tel:1-951-572-2633> IM: IRC Freenode/OFTC: thingee 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> <https://www.redhat.com> <https://redhat.com/summit>

1 1

RBD pool damaged, repair options?
by Robert Sander 13 Aug '20

13 Aug '20

Hi, a customer lost 5 OSDs at the same time (and replaced them with new disks before we could do anything…). 4 PGs were incomplete but could be repaired with ceph-objectstore-tool. The cluster itself is healthy again. Now some RBDs are missing. They are still listed in the rbd_directory object but cannot be access with e.g. rbd info. rbd ls says: no such file or directory I assume that is because the associated rbd_id.name object has been lost. I tried to create this object by means of "rados put" but still no luck: 2020-08-13 11:47:27.557 7fa1e2ffd700 -1 librbd::image::OpenRequest: failed to retrieve image id: (5) Input/output error 2020-08-13 11:47:27.557 7fa1e27fc700 -1 librbd::ImageState: 0x55b9ebb75b10 failed to open image: (5) Input/output error 2020-08-13 11:47:27.557 7fa1e27fc700 -1 librbd::io::AioCompletion: 0x55b9ebadd3a0 fail: (5) Input/output error rbd: error opening vm-501-disk-2: (5) Input/output error What are the options to repair an RBD pool so that at least the RBDs (most data objects are still there) are available again. Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

Approach Epson Customer Service For Fixing Network Problems.
by mary smith 13 Aug '20

13 Aug '20

Yes, you are not only allowed to approach Epson Customer Service but also fetch the instant technical backing directly from the professionals. If you want to avail the quickest remedy along with the supervision regarding the problems and hurdles, pertaining to HP printer network, you should get the free assistance at anytime. https://www.epsonprintersupportpro.net/

1 0

Block Network IP Of Wireless HP Printer Using HP Support Assistant.
by mary smith 13 Aug '20

13 Aug '20

Yes, you should! If you are one of those who are have a word with the professionals who are capable of resolving the whole host of network problems. You can make use of HP Support Assistant who will solve the whole host of wireless printer problems. https://www.amiytech.com/hp-support-assistant/

1 0

Heavy rocksdb activity in newly added osd
by Jerry Pu 13 Aug '20

13 Aug '20

Hi All We had a cluster (v13.2.4) with 32 osds in total. At first, an osd (osd.18) in cluster was down. So, we tried to remove the osd and added a new one (osd.32) with new ID. We unplugged the disk (osd.18) and plugged in a new disk in the same slot and add osd.32 into cluster. Then, osd.32 was booting, but, we found it takes much time (around 18 mins) for the osd to change to up state. Diving into osd.32 logs, we see that there is much rocksdb activity before osd.32 change to up state. Can anyone explain why this happened or give me any advice about how to prevent from this. Thanks. [osd.32 log] 2020-08-03 15:36:58.852 7f88021fa1c0 0 osd.32 0 done with init, starting boot process 2020-08-03 15:36:58.852 7f88021fa1c0 1 osd.32 0 start_boot 2020-08-03 15:36:58.854 7f87db02b700 -1 osd.32 0 waiting for initial osdmap 2020-08-03 15:36:58.855 7f87e4ba0700 -1 osd.32 0 failed to load OSD map for epoch 22010, got 0 bytes 2020-08-03 15:36:58.955 7f87e0836700 0 osd.32 22011 crush map has features 283675107524608, adjusting msgr requires for clients 2020-08-03 15:36:58.955 7f87e0836700 0 osd.32 22011 crush map has features 283675107524608 was 288232575208792577, adjusting msgr requires for mons *2020-08-03 15:36:58.955* 7f87e0836700 0 osd.32 22011 crush map has features 720859615486820352, adjusting msgr requires for osds 2020-08-03 15:37:31.182 7f87e1037700 4 rocksdb: [/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/db_impl_write.cc:1346] [default] New memtable created with log file: #16. Immutable memtables: 0. 2020-08-03 15:37:31.285 7f87e8045700 4 rocksdb: (Original Log Time 2020/08/03-15:37:31.183995) [/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/db_impl_compaction_flush.cc:1396] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2020-08-03 15:37:31.285 7f87e8045700 4 rocksdb: [/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/flush_job.cc:300] [default] [JOB 3] Flushing memtable with next log file: 16 -------- lots of rocksdb activity--------- 2020-08-03 15:54:21.704 7f87e8045700 4 rocksdb: (Original Log Time 2020/08/03-15:54:21.705680) [/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/memtable_list.cc:397] [default] Level-0 commit table #112: memtable #1 done 2020-08-03 15:54:21.704 7f87e8045700 4 rocksdb: (Original Log Time 2020/08/03-15:54:21.705704) EVENT_LOG_v1 {"time_micros": 1596441261705697, "job": 51, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 3, 0, 0, 0, 0, 0], "immutable_memtables": 0} 2020-08-03 15:54:21.704 7f87e8045700 4 rocksdb: (Original Log Time 2020/08/03-15:54:21.705721) [/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/db_impl_compaction_flush.cc:172] [default] Level summary: base level 1 max bytes base 268435456 files[1 3 0 0 0 0 0] max score 0.75 *2020-08-03 15:54:38.567* 7f87e0836700 1 osd.32 502096 state: booting -> active 2020-08-03 15:54:38.567 7f87d5820700 1 osd.32 pg_epoch: 502096 pg[1.17e( empty local-lis/les=0/0 n=0 ec=11627/16 lis/c 501703/501703 les/c/f 501704/501704/0 502096/502096/502096) [32,26,28] r=0 lpr=502096 pi=[501703,502096)/1 crt=0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary Best Jerry

1 0

Can I install HP envy 4520 drivers correctly?
by Vin Diesel 13 Aug '20

13 Aug '20

HP envy 4520 wireless printer is one of the most popular brands, which is specially known for its printing services. It is the best product in the features and functions, so millions of users prefer it for the best printing performance. Due to some technical problems, my HP 4520 drivers are fully corrupted, so I want to install the newest version of HP envy 4520 drivers successfully. As per my technical experience, I am trying my skills to install HP envy 4520 printer driver. I am failing to complete installation process. So anyone can help me to install printer driver appropriately. https://www.hpprintersupportpro.com/blog/troubleshoot-hp-printer-envy-4520-…

1 0

Can I install HP envy 4520 drivers correctly?
by Vin Diesel 13 Aug '20

13 Aug '20

HP envy 4520 wireless printer is one of the most popular brands, which is specially known for its printing services. It is the best product in the features and functions, so millions of users prefer it for the best printing performance. Due to some technical problems, my HP 4520 drivers are fully corrupted, so I want to install the newest version of [url=https://www.hpprintersupportpro.com/blog/troubleshoot-hp-printer-envy-4… envy 4520 drivers[/url] successfully. As per my technical experience, I am trying my skills to install HP envy 4520 printer driver. I am failing to complete installation process. So anyone can help me to install printer driver appropriately.

1 0

Ceph not warning about clock skew on an OSD-only host?
by Matthew Vernon 13 Aug '20

13 Aug '20

Hi, Our production cluster runs Luminous. Yesterday, one of our OSD-only hosts came up with its clock about 8 hours wrong(!) having been out of the cluster for a week or so. Initially, ceph seemed entirely happy, and then after an hour or so it all went South (OSDs start logging about bad authenticators, I/O pauses, general sadness). I know clock sync is important to Ceph, so "one system is 8 hours out, Ceph becomes sad" is not a surprise. It is perhaps a surprise that the OSDs were allowed in at all... What _is_ a surprise, though, is that at no point in all this did Ceph raise a peep about clock skew. Normally it's pretty sensitive to this - our test cluster has had clock skew complaints when a mon is only slightly out, and here we had a node 8 hours wrong. Is there some oddity like Ceph not warning on clock skew for OSD-only hosts? or an upper bound on how high a discrepency it will WARN about? Regards, Matthew example output from mid-outage: root@sto-3-1:~# ceph -s cluster: id: 049fc780-8998-45a8-be12-d3b8b6f30e69 health: HEALTH_ERR 40755436/2702185683 objects misplaced (1.508%) Reduced data availability: 20 pgs inactive, 20 pgs peering Degraded data redundancy: 367431/2702185683 objects degraded (0.014%), 4549 pgs degraded 481 slow requests are blocked > 32 sec. Implicated osds 188,284,795,1278,1981,2061,2648,2697 644 stuck requests are blocked > 4096 sec. Implicated osds 22,31,33,35,101,116,120,130,132,140,150,159,201,211,228,263,327,541,561,566,585,589,636,643,649,654,743,785,790,806,865,1037,1040,1090,1100,1104,1115,1134,1135,1166,1193,1275,1277,1292,1494,1523,1598,1638,1746,2055,2069,2191,2210,2358,2399,2486,2487,2562,2589,2613,2627,2656,2713,2720,2837,2839,2863,2888,2908,2920,2928,2929,2947,2948,2963,2969,2972 [...] -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

3 2

fio rados ioengine
by Frank Ritchie 12 Aug '20

12 Aug '20

Hi all, Does anyone know if using fio with the rados ioengine will produce similar results to rados bench, i.e. can the 2 be used interchangeably? thx Frank

1 0

RGW unable to delete a bucket
by Andrei Mikhailovsky 12 Aug '20

12 Aug '20

Hi I am trying to delete a bucket using the following command: # radosgw-admin bucket rm --bucket=<bucket-name> --purge-objects However, in console I get the following messages. About 100+ of those messages per second. 2020-08-04T17:11:06.411+0100 7fe64cacf080 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 The command has been running for about 35 days days and it still hasn't finished. The size of the bucket is under 1TB for sure. Probably around 500GB. I have recently removed about a dozen of old buckets without any issues. It's this particular bucket that is being very stubborn. Anything I can do to remove it, including it's objects and any orphans it might have? Thanks Andrei

3 3

DocuBetter Meeting Today 1630 UTC
by John Zachary Dover 12 Aug '20

12 Aug '20

There is a general documentation meeting called the "DocuBetter Meeting", and it is held every two weeks. The next DocuBetter Meeting will be on 12 Aug 2020 at 0830 PDT, and will run for thirty minutes. Everyone with a documentation-related request or complaint is invited. The meeting will be held here: https://bluejeans.com/908675367 This meeting will cover the reorganization of the Ceph website as well as Zac's recently-developed workflow that aims to make it possible to move the good ideas from stale or poorly-formed documentation PRs into the documentation more quickly. Send documentation-related requests and complaints to me by replying to this email and CCing me at zac.dover(a)gmail.com. The next DocuBetter meeting is scheduled for: 12 Aug 2020 0830 PDT 12 Aug 2020 1630 UTC 13 Aug 2020 0230 AEST Etherpad: https://pad.ceph.com/p/Ceph_Documentation Zac's docs whiteboard: https://pad.ceph.com/p/docs_whiteboard Meeting: https://bluejeans.com/908675367

1 0

Meaning of the "tag" key in bucket metadata
by Wesley Dillingham 12 Aug '20

12 Aug '20

Recently we encountered an instance of bucket corruption of two varieties. One in which the bucket metadata was missing and another in which the bucket.instance metadata was missing for various buckets. We have seemingly been successful in restoring the metadata by reconstructing it from the remaining pieces of metadata and injecting it using "radosgw-admin metadata put" for both bucket and bucket.instance metadata One piece of information that we could not determine was "tag" "ver": { "tag": "_iIMyX8XLf0HSTciEsrgLA7j", "ver": 1 }, and so we tried reusing the tag in bucket.instance for the bucket metadata and also spoofed the tag value to something random. It appears in both situations the bucket's functionality was restored. I am however uncertain of the function of this "tag" key and what situation I may be exposing myself to by reusing or spoofing its value. Respectfully, *Wes Dillingham* wes(a)wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>

1 0

Geek squad services
by hirkumispe＠enayu.com 12 Aug '20

12 Aug '20

Are you looking for Geek Squad Tech Support, Visit Geek Tech Support now, to compare the offers for Geek Squad Tech Support. All channel support plans. https://thegethelp.com/geek-squad-tech-support/ Get your devices protected under our geek protection plan. Check out our detailed guide on how egeek protection plan is better than the geek squad protection plan https://egeekonline.com/protection-plan/

1 0

Get World-Class Programming Assignment Help at a Reasonable budget.
by keithmorris081＠gmail.com 12 Aug '20

12 Aug '20

Get it completed from the Top notch Assignment Writers for the Students across Universities across various countries. With the best in class turnaround time and also avail amazing value-added service, And Get in touch with the Best Programming Assignment Help who are very talented in providing solutions on various topics on subjects. You Read it Right! We are here to help you with your queries on your assignments. We are providing you with a world-class and Highly Talented Assignment expert. These Assignment Writers who are capable of doing any writing services, and many other writing services such as Assignment Writing Services, Assignment Help for various subjects such as Engineering Assignment, Statistics for the students across Australia, and also for students across UK, USA, Malaysia and various other countries. https://www.myassignmentservices.com/programming-assignment-help.html

2 1

professional essay writing help
by keiratayor＠gmail.com 12 Aug '20

12 Aug '20

This professional help with writing essays for college not only provides flawless essays to the students but it also takes several initiatives to make the service comprehensive and inclusive for the students. Accordingly, the expert essay writer has published numerous history essay samples on the website. https://essayassignmenthelp.com.au/essay-writing-help.html

1 0

Speeding up reconnection
by William Edwards 12 Aug '20

12 Aug '20

Hello, When connection is lost between kernel client, a few things happen: 1. Caps become stale: Aug 11 11:08:14 admin-cap kernel: [308405.227718] ceph: mds0 caps stale 2. MDS evicts client for being unresponsive: MDS log: 2020-08-11 11:12:08.923 7fd1f45ae700 0 log_channel(cluster) log [WRN] : evicting unresponsive client admin-cap.cf.ha.cyberfusion.cloud:DB0001-cap (144786749), after 300.978 seconds Client log: Aug 11 11:12:11 admin-cap kernel: [308643.051006] ceph: mds0 hung 3. Socket is closed: Aug 11 11:22:57 admin-cap kernel: [309289.192705] libceph: mds0 [fdb7:b01e:7b8e:0:10:10:10:1]:6849 socket closed (con state OPEN) I am not sure whether the kernel client or MDS closes the connection. I think the kernel client does so, because nothing is logged at the MDS side at 11:22:57 4. Connection is reset by MDS: MDS log: 2020-08-11 11:22:58.831 7fd1f9e49700 0 --1- [v2:[fdb7:b01e:7b8e:0:10:10:10:1]:6800/3619156441,v1:[fdb7:b01e:7b8e:0:10:10:10:1]:6849/3619156441] >> v1:[fc00:b6d:cfc:951::7]:0/133007863 conn(0x55bfaf1c2880 0x55c16cb47000 :6849 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept we reset (peer sent cseq 1), sending RESETSESSION Client log: Aug 11 11:22:58 admin-cap kernel: [309290.058222] libceph: mds0 [fdb7:b01e:7b8e:0:10:10:10:1]:6849 connection reset 5. Kernel client reconnects: Aug 11 11:22:58 admin-cap kernel: [309290.058972] ceph: mds0 closed our session Aug 11 11:22:58 admin-cap kernel: [309290.058973] ceph: mds0 reconnect start Aug 11 11:22:58 admin-cap kernel: [309290.069979] ceph: mds0 reconnect denied Aug 11 11:22:58 admin-cap kernel: [309290.069996] ceph: dropping file locks for 000000006a23d9dd 1099625041446 Aug 11 11:22:58 admin-cap kernel: [309290.071135] libceph: mds0 [fdb7:b01e:7b8e:0:10:10:10:1]:6849 socket closed (con state NEGOTIATING) Question: As you can see, there's 10 minutes between losing the connection and the reconnection attempt (11:12:08 - 11:22:58). I could not find any settings related to the period after which reconnection is attempted. I would like to change this value from 10 minutes to something like 1 minute. I also tried searching the Ceph docs for the string '600' (10 minutes), but did not find anything useful. Hope someone can help. Environment details: Client kernel: 4.19.0-10-amd64 Ceph version: ceph version 14.2.9 (bed944f8c45b9c98485e99b70e11bbcec6f6659a) nautilus (stable) Met vriendelijke groeten, William Edwards

3 3

What The Benefit Of Choosing Epson Customer Service?
by mary smith 12 Aug '20

12 Aug '20

All together get profited by picking the right and trustworthy Epson Customer Service specialist co-ops, it is firmly prescribed to have a solid examination before engaging with any of such specialist organizations as there are a few phony organizations are likewise there professing to be the best yet really not. https://www.epsonprintersupportpro.net/

1 0

Get the opportunity To Be Acquainted With HP Support Assistant In A Proper Manner.
by mary smith 12 Aug '20

12 Aug '20

So as to be familiar with the HP Support Assistant in an appropriate way, you ought not leave any stone unturned in moving toward the guaranteed printer specialists who will expertly help you the specific investigating answer for your issues in a powerful way. You can move toward them by simply utilizing distinctive advantageous methods at whenever. https://www.amiytech.com/hp-support-assistant/

1 0

It takes long time for a newly added osd booting to up state due to heavy rocksdb activity
by Jerry Pu 12 Aug '20

12 Aug '20

Hi All We had a cluster (v13.2.4) with 32 osds in total. At first, an osd (osd.18) in cluster was down. So, we tried to remove the osd and added a new one (osd.32) with new ID. We unplugged the disk (osd.18) and plugged in a new disk in the same slot and add osd.32 into cluster. Then, osd.32 was booting, but, we found it takes much time (around 18 mins) for the osd to change to up state. Diving into osd.32 logs, we see that there is much rocksdb activity before osd.32 change to up state. Can anyone explain why this happened or give me any advice about how to prevent from this. Thanks. [osd.32 log] 2020-08-03 15:36:58.852 7f88021fa1c0 0 osd.32 0 done with init, starting boot process 2020-08-03 15:36:58.852 7f88021fa1c0 1 osd.32 0 start_boot 2020-08-03 15:36:58.854 7f87db02b700 -1 osd.32 0 waiting for initial osdmap 2020-08-03 15:36:58.855 7f87e4ba0700 -1 osd.32 0 failed to load OSD map for epoch 22010, got 0 bytes 2020-08-03 15:36:58.955 7f87e0836700 0 osd.32 22011 crush map has features 283675107524608, adjusting msgr requires for clients 2020-08-03 15:36:58.955 7f87e0836700 0 osd.32 22011 crush map has features 283675107524608 was 288232575208792577, adjusting msgr requires for mons *2020-08-03 15:36:58.955* 7f87e0836700 0 osd.32 22011 crush map has features 720859615486820352, adjusting msgr requires for osds 2020-08-03 15:37:31.182 7f87e1037700 4 rocksdb: [/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/db_impl_write.cc:1346] [default] New memtable created with log file: #16. Immutable memtables: 0. 2020-08-03 15:37:31.285 7f87e8045700 4 rocksdb: (Original Log Time 2020/08/03-15:37:31.183995) [/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/db_impl_compaction_flush.cc:1396] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2020-08-03 15:37:31.285 7f87e8045700 4 rocksdb: [/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/flush_job.cc:300] [default] [JOB 3] Flushing memtable with next log file: 16 -------- lots of rocksdb activity--------- 2020-08-03 15:54:21.704 7f87e8045700 4 rocksdb: (Original Log Time 2020/08/03-15:54:21.705680) [/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/memtable_list.cc:397] [default] Level-0 commit table #112: memtable #1 done 2020-08-03 15:54:21.704 7f87e8045700 4 rocksdb: (Original Log Time 2020/08/03-15:54:21.705704) EVENT_LOG_v1 {"time_micros": 1596441261705697, "job": 51, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 3, 0, 0, 0, 0, 0], "immutable_memtables": 0} 2020-08-03 15:54:21.704 7f87e8045700 4 rocksdb: (Original Log Time 2020/08/03-15:54:21.705721) [/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/db_impl_compaction_flush.cc:172] [default] Level summary: base level 1 max bytes base 268435456 files[1 3 0 0 0 0 0] max score 0.75 *2020-08-03 15:54:38.567* 7f87e0836700 1 osd.32 502096 state: booting -> active 2020-08-03 15:54:38.567 7f87d5820700 1 osd.32 pg_epoch: 502096 pg[1.17e( empty local-lis/les=0/0 n=0 ec=11627/16 lis/c 501703/501703 les/c/f 501704/501704/0 502096/502096/502096) [32,26,28] r=0 lpr=502096 pi=[501703,502096)/1 crt=0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary Best Jerry

1 0

v14.2.11 Nautilus released
by Abhishek Lekshmanan 11 Aug '20

11 Aug '20

We're happy to announce the availability of the eleventh release in the Nautilus series. This release brings a number of bugfixes across all major components of Ceph. We recommend that all Nautilus users upgrade to this release. Notable Changes --------------- * RGW: The `radosgw-admin` sub-commands dealing with orphans -- `radosgw-admin orphans find`, `radosgw-admin orphans finish`, `radosgw-admin orphans list-jobs` -- have been deprecated. They have not been actively maintained and they store intermediate results on the cluster, which could fill a nearly-full cluster. They have been replaced by a tool, currently considered experimental, `rgw-orphan-list`. * Now when noscrub and/or nodeep-scrub flags are set globally or per pool, scheduled scrubs of the type disabled will be aborted. All user initiated scrubs are NOT interrupted. * Fixed a ceph-osd crash in _committed_osd_maps when there is a failure to encode the first incremental map. issue#46443: https://github.com/ceph/ceph/pull/46443 For the detailed changelog please refer to the blog entry at https://ceph.io/releases/v14-2-11-nautilus-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.11.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: f7fdb2f52131f54b891a2ec99d8205561242cdaf -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)

1 0

ceph orch host rm seems to just move daemons out of cephadm, not remove them
by pixel fairy 11 Aug '20

11 Aug '20

made a cluster of 2 osd hosts, and one temp monitor. then added another osd host and did a "ceph orch host rm tempmon". this all in vagrant (libvirt), with the generic/ubuntu2004 box. INFO:cephadm:Inferring fsid 5426a59e-db33-11ea-8441-b913b695959d INFO:cephadm:Using recent ceph image ceph/ceph:v15 cluster: id: 5426a59e-db33-11ea-8441-b913b695959d health: HEALTH_WARN 2 stray daemons(s) not managed by cephadm 1 stray host(s) with 2 daemon(s) not managed by cephadm added 2 more osd hosts, and ceph -s gave me this, services: mon: 6 daemons, quorum ceph5,ceph4,tempmon,ceph3,ceph2,ceph1 (age 33m) mgr: ceph5.erdofb(active, since 82m), standbys: tempmon.xkrlmm, ceph3.xjuecs osd: 15 osds: 15 up (since 33m), 15 in (since 33m) my guess is cephadm wanted 5 managed mons, so did that, but still never removed the removed mon on the removed host. its still up. this is just a vagrant file. so i have two questions. 1. how do you remove that other host and its daemons from the cluster? 2. how would you recover from a host being destroyed? p.s. tried google Your search - "ceph orch host rm" "stray daemons(s) not manage by cephadm" - did not match any documents.

1 1

Announcing go-ceph v0.5.0
by John Mulligan 11 Aug '20

11 Aug '20

I'm happy to announce the another release of the go-ceph API bindings. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.5.0 The bindings aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. These API bindings require the use of cgo. There are already a few consumers of this library in the wild, including the ceph-csi project. Specific questions, comments, bugs etc are best directed at our github issues tracker. --- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com

1 0

pgs not deep scrubbed in time - false warning?
by Dirk Sarpe 11 Aug '20

11 Aug '20

Hi, since some time (I think upgrade to nautilus) we get X pgs not deep scrubbed in time I deep-scrubbed the pgs when the error occurred and expected the cluster to recover over time, but no such luck. The warning comes up again and again. In our spinning rust cluster we allow deep scrubbing only from 19:00 to 06:00 and changed the deep scrub interval to 28 days (detailed osd scrub config below). Looking at the pgs which are supposedly not deep scrubbed in time, reveals that their 28 day period is not over yet, example: # date && ceph health detail | awk '$1=="pg" { print $0 }' Di 11. Aug 09:32:17 CEST 2020 pg 0.787 not deep-scrubbed since 2020-07-30 03:08:24.899264 pg 0.70c not deep-scrubbed since 2020-07-30 02:45:08.989329 pg 0.6c1 not deep-scrubbed since 2020-07-30 03:01:15.199496 pg 13.3 not deep-scrubbed since 2020-07-30 03:29:54.536825 pg 0.d9 not deep-scrubbed since 2020-07-30 03:12:34.503586 pg 0.41a not deep-scrubbed since 2020-07-30 03:01:23.514582 pg 0.490 not deep-scrubbed since 2020-07-30 03:05:45.616100 I wonder if I have missed or messed up some parameter that could cause this. # ceph daemon osd.40 config show | grep scrub "mds_max_scrub_ops_in_progress": "5", "mon_scrub_inject_crc_mismatch": "0.000000", "mon_scrub_inject_missing_keys": "0.000000", "mon_scrub_interval": "86400", "mon_scrub_max_keys": "100", "mon_scrub_timeout": "300", "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", "mon_warn_pg_not_scrubbed_ratio": "0.500000", "osd_debug_deep_scrub_sleep": "0.000000", "osd_deep_scrub_interval": "2419200.000000", "osd_deep_scrub_keys": "1024", "osd_deep_scrub_large_omap_object_key_threshold": "200000", "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", "osd_deep_scrub_randomize_ratio": "0.150000", "osd_deep_scrub_stride": "1048576", "osd_deep_scrub_update_digest_min_age": "7200", "osd_max_scrubs": "1", "osd_op_queue_mclock_scrub_lim": "0.001000", "osd_op_queue_mclock_scrub_res": "0.000000", "osd_op_queue_mclock_scrub_wgt": "1.000000", "osd_requested_scrub_priority": "120", "osd_scrub_auto_repair": "false", "osd_scrub_auto_repair_num_errors": "5", "osd_scrub_backoff_ratio": "0.660000", "osd_scrub_begin_hour": "19", "osd_scrub_begin_week_day": "0", "osd_scrub_chunk_max": "25", "osd_scrub_chunk_min": "5", "osd_scrub_cost": "52428800", "osd_scrub_during_recovery": "false", "osd_scrub_end_hour": "6", "osd_scrub_end_week_day": "7", "osd_scrub_interval_randomize_ratio": "0.500000", "osd_scrub_invalid_stats": "true", "osd_scrub_load_threshold": "0.500000", "osd_scrub_max_interval": "604800.000000", "osd_scrub_max_preemptions": "5", "osd_scrub_min_interval": "172800.000000", "osd_scrub_priority": "5", "osd_scrub_sleep": "0.100000", ceph version 14.2.10 My ad hoc helper script #!/bin/bash # gently deep scrub all pgs which failed to deep scrub in set period ceph health detail | awk '$1=="pg" { print $2 } ' | while read -r pg do ceph pg deep-scrub "$pg" sleep 60 done Cheers, Dirk

1 1

Get instant help for Cash App Refund issues.
by mary smith 11 Aug '20

11 Aug '20

Do you Cash App Refund on your cash app balance or bank account? If yes then connect with the technical team as they ensure quick tech support services, You’ll be guided with precise solutions for the refund process. Just call the technical experts and seek their assistance on this. You can call them at any time of the day for support. https://www.customercare-email.com/blog/how-to-refund-a-payment-on-your-cas…

1 0

Fix Cash App transfer error? Contact Cash.app/help for help.
by mary smith 11 Aug '20

11 Aug '20

One of the app’s primary function is transfer and making payments. But if there’s a tech error that’s not encouraging you to do that, then you can watch some tech videos on Youtube for assistance or you can call the Cash.app/help number for the assistance and relay your problem to the customer care rep to get a solution to fix the issue. https://www.cash-app-customer-service.com/

1 0

Deleterious effects OSD queue
by João Victor Mafra 10 Aug '20

10 Aug '20

Hi folks, Regarding OSD operation queues (shards), I found this in the documentation (https://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/) "A lower number of shards will increase the impact of the mClock queues, but may have other deleterious effects". What are these deleterious effects? Why is it so bad to use just one queue for example? Is there any kind of bottleneck or costly procedures in these queues?

1 0

EntityAddress format in ceph ssd blacklist commands
by Void Star Nill 10 Aug '20

10 Aug '20

Hi, I want to understand the <EntityAddress> format for `ceph osd blacklist` commands. The documentation just says it's the address. But I am not sure if it can just be the host IP address or anything else. What does *:0/* *3710147553* represent in the following output? $ ceph osd blacklist ls listed 1 entries127.0.0.1:0/3710147553 2018-03-19 11:32:24.716146 $ ceph osd blacklist rm 127.0.0.1:0/3710147553 un-blacklisting 127.0.0.1:0/3710147553 Regards, Shridhar

2 4

ceph rbd iscsi gwcli Non-existent images
by Steven Vacaroaia 10 Aug '20

10 Aug '20

Hi, I would appreciate any help/hints to solve this issue iscis (gwcli) cannot see the images anymore This configuration worked fine for many months What changed was that ceph is "nearly full" I am in the process of cleaning it up ( by deleting objects from one of the pools) and I do see reads and writes on the cluster as well as images info so not sure what gwcli does not like ( targetcli ls not working either - just froze ) Below some info ceph version ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable) gwcli --version gwcli - 2.7 ceph osd dump | grep ratio full_ratio 0.96 backfillfull_ratio 0.92 nearfull_ratio 0.9 [root@osd02 ~]# rbd -p rbd info rep01 rbd image 'rep01': size 7 TiB in 1835008 objects order 22 (4 MiB objects) id: 15b366b8b4567 block_name_prefix: rbd_data.15b366b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Nov 1 15:57:52 2018 [root@osd02 ~]# rbd -p rbd info vmware01 rbd image 'vmware01': size 6 TiB in 1572864 objects order 22 (4 MiB objects) id: 16d3f6b8b4567 block_name_prefix: rbd_data.16d3f6b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Nov 29 13:56:28 2018 [root@osd02 ~]# ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 33 TiB 7.5 TiB 25 TiB 77.16 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS cephfs_metadata 22 173 MiB 0.01 1.4 TiB 469 cephfs_data 23 1.7 TiB 69.78 775 GiB 486232 rbd 24 11 TiB 93.74 775 GiB 2974077 [root@osd02 ~]# ceph health detail HEALTH_ERR 2 nearfull osd(s); 2 pool(s) nearfull; Module 'prometheus' has failed: IOError("Port 9283 not free on '10.10.35.20'",) OSD_NEARFULL 2 nearfull osd(s) osd.12 is near full osd.17 is near full POOL_NEARFULL 2 pool(s) nearfull pool 'cephfs_data' is nearfull pool 'rbd' is nearfull gwcli /iscsi-target...nner-21faa413> info Client Iqn .. iqn.1998-01.com.vmware:banner-21faa413 Ip Address .. Alias .. Logged In .. Auth - chap .. cephuser/PASSWORD Group Name .. Luns - rbd.rep01 .. lun_id=0 - rbd.vmware01 .. lun_id=1 osd02 journal: client update failed on iqn.1998-01.com.vmware:banner-21faa413 : Non-existent images ['rbd.vmware01'] requested for iqn.1998-01.com.vmware:banner-21faa413 Aug 7 14:15:39 osd02 journal: 127.0.0.1 - - [07/Aug/2020 14:15:39] "PUT /api/_clientlun/iqn.1998-01.com.vmware:banner-21faa413 HTTP/1.1" 500 - Aug 7 14:15:39 osd02 journal: _clientlun change on 127.0.0.1 failed with 500 Aug 7 14:15:39 osd02 journal: 127.0.0.1 - - [07/Aug/2020 14:15:39] "DELETE /api/clientlun/iqn.1998-01.com.vmware:banner-21faa413 HTTP/1.1" 500 -

2 3

Re: RGW unable to delete a bucket
by Matt Benjamin 10 Aug '20

10 Aug '20

Hi Folks, I don't know of a downstream issue that looks like this, and we've upstreamed every fix for bucket listing and cleanup we have. We are pursuing a space leak believed to arise in "radosgw-admin bucket rm --purge-objects" but not a non-terminating listing. The only upstream release not planned to get a backport of orphans list tools is Luminous. I thought backport to Octopus was already done by the backport team? regards, Matt On Thu, Aug 6, 2020 at 2:40 PM EDH - Manuel Rios <mriosfer(a)easydatahost.com> wrote: > > You'r not the only one affected by this issue > > As far as i know several huge companies hitted this bug too, but private patches or tools are not public released. > > This is caused for the a resharding process during upload in previous versions. > > Workarround for us.: > > - Delete objects of the bucket at rados level. > - Delete the index file of the bucket. > > Pray to god to not happen again. > > Still pending backporting to Nautilus of the new experimental tool to find orphans in RGW > > Maybe @Matt Benjamin can give us and ETA for get ready that tool backported... > > Regards > > > > -----Mensaje original----- > De: Andrei Mikhailovsky <andrei(a)arhont.com> > Enviado el: jueves, 6 de agosto de 2020 13:55 > Para: ceph-users <ceph-users(a)ceph.io> > Asunto: [ceph-users] Re: RGW unable to delete a bucket > > BUMP... > > > ----- Original Message ----- > > From: "Andrei Mikhailovsky" <andrei(a)arhont.com> > > To: "ceph-users" <ceph-users(a)ceph.io> > > Sent: Tuesday, 4 August, 2020 17:16:28 > > Subject: [ceph-users] RGW unable to delete a bucket > > > Hi > > > > I am trying to delete a bucket using the following command: > > > > # radosgw-admin bucket rm --bucket=<bucket-name> --purge-objects > > > > However, in console I get the following messages. About 100+ of those > > messages per second. > > > > 2020-08-04T17:11:06.411+0100 7fe64cacf080 1 > > RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket > > listing requires read #1 > > > > > > The command has been running for about 35 days days and it still > > hasn't finished. The size of the bucket is under 1TB for sure. Probably around 500GB. > > > > I have recently removed about a dozen of old buckets without any > > issues. It's this particular bucket that is being very stubborn. > > > > Anything I can do to remove it, including it's objects and any orphans > > it might have? > > > > > > Thanks > > > > Andrei > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309

2 1

DocuBetter Meeting this week -- 12 Aug 2020 0830 PDT
by John Zachary Dover 10 Aug '20

10 Aug '20

There is a general documentation meeting called the "DocuBetter Meeting", and it is held every two weeks. The next DocuBetter Meeting will be on 12 Aug 2020 at 0830 PDT, and will run for thirty minutes. Everyone with a documentation-related request or complaint is invited. The meeting will be held here: https://bluejeans.com/908675367 This meeting will cover the reorganization of the Ceph website as well as Zac's recently-developed workflow that aims to make it possible to move the good ideas from stale or poorly-formed documentation PRs into the documentation more quickly. Send documentation-related requests and complaints to me by replying to this email and CCing me at zac.dover(a)gmail.com. The next DocuBetter meeting is scheduled for: 12 Aug 2020 0830 PDT 12 Aug 2020 1630 UTC 13 Aug 2020 0230 AEST Etherpad: https://pad.ceph.com/p/Ceph_Documentation Zac's docs whiteboard: https://pad.ceph.com/p/docs_whiteboard Meeting: https://bluejeans.com/908675367

1 0

RGW 14.2.10 Regresion? ordered bucket listing requires read #1
by EDH - Manuel Rios 10 Aug '20

10 Aug '20

Hi, We got our cluster updated to the lastest versión 14.2.10 Checking rgw logs after 14.2.10 upgrade 2020-08-10 10:21:49.186 7f74cd7db700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.188 7f75eca19700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.189 7f7479f34700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.193 7f74d77ef700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.196 7f753c0b8700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.197 7f74cd7db700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 2020-08-10 10:21:49.197 7f745d6fb700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 Looks like this -> https://tracker.ceph.com/issues/42358 Regards Manuel

1 0