- ceph-users - lists.ceph.io

Setting rbd_default_data_pool through the config store
by Wido den Hollander 30 Jul '20

30 Jul '20

Hi, I'm trying to have clients read the 'rbd_default_data_pool' config option from the config store when creating a RBD image. This doesn't seem to work and I'm wondering if somebody knows why. I tried: $ ceph config set client rbd_default_data_pool rbd-data $ ceph config set global rbd_default_data_pool rbd-data They both show up under: $ ceph config dump However, newly created RBD images with the 'rbd' CLI tool do not use the data pool. If I set this in ceph.conf it works: [client] rbd_default_data_pool = rbd-data Somehow librbd isn't fetching these configuration options. Any hints on how to get this working? The end result is that libvirt (which doesn't read ceph.conf) should also be able to create RBD images with a different data pool. Wido

2 6

Usable space vs. Overhead
by David Orman 29 Jul '20

29 Jul '20

I'm having a hard time understanding the EC usable space vs. raw. https://ceph.io/geen-categorie/ceph-erasure-coding-overhead-in-a-nutshell/ indicates "nOSD * k / (k+m) * OSD Size" is how you calculate usable space, but that's not lining up with what i'd expect just from k data chunks + m parity chunks. So, for example, k=4, m=2. you'd expect every 4 byte object written would consume 6 bytes, so 50% overhead. however, the prior formula in a 7 server cluster, using 4+2 encoding, would indicate 66.67% usable capacity vs. raw storage. What am I missing here?

4 7

Re: S3 bucket lifecycle not deleting old objects
by Robin H. Johnson 29 Jul '20

29 Jul '20

On Tue, Jul 28, 2020 at 01:28:14PM +0000, Alex Hussein-Kershaw wrote: > Hello, > > I have a problem that old versions of S3 objects are not being deleted. Can anyone advise as to why? I'm using Ceph 14.2.9. How many objects are in the bucket? If it's a lot, then you may run into RGW's lifecycle performance limitations: listing each bucket is a very slow operation for lifecycle prior to improvements make in later versions (Octopus with maybe a backport to Nautilius?) If the bucket doesn't have a lot of operations, you could try running the 'radosgw-admin lc process' directly, with debug logging, and see where it gets bogged down. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2(a)gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

2 1

How to change toner in sharp printer
by hirkumispe＠enayu.com 29 Jul '20

29 Jul '20

Don't know how to change toner cartridge on your sharp printer? Check our this step by step guide to learn how to replace toner in sharp printer https://printersetup.org/change-toner-in-sharp-printer/

1 0

How to contact geek squad online
by hirkumispe＠enayu.com 29 Jul '20

29 Jul '20

Fix technical breakdown of all your electronics and appliances at Geek Squad Support. Reach the certified experts at Geek Squad Support for fixing any kind of technical bug with your devices. Best of services and assistance assured at support. https://geekstechs.org/geek-squad-support/ Remove the technical glitches and flaws of your electronic appliances with Geek Squad Online Support. Avail the best online support services for fixing your electronic appliance. Reach the experts at Geek Squad Online Support with 24/7 availability to avail best solutions for your troublesome devices. https://geek-customer-care.com/geek-squad-online-support.html

1 0

Stuck removing osd with orch
by Ml Ml 29 Jul '20

29 Jul '20

Hello, yesterday i did: ceph osd purge 32 --yes-i-really-mean-it I also started to upgrade: ceph orch upgrade start --ceph-version 15.2.4 It seems its really gone: ceph osd crush remove osd.32 => device 'osd.32' does not appear in the crush map ceph orch ps: osd.32 ceph01 error 4h ago 2h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> root@ceph01:~# ceph orch osd rm status NAME HOST PGS STARTED_AT osd.32 ceph01 n/a 2020-07-29 09:50:11.026643 (some hours ago) root@ceph01:~# ceph health detail HEALTH_ERR Module 'cephadm' has failed: auth get failed: failed to find osd.32 in keyring retval: -2; Low space hindering backfill (add storage if this doesn't resolve itself): 105 pgs backfill_toofull; Degraded data redundancy: 9882/33167949 objects degraded (0.030%), 1 pg degraded, 1 pg undersized [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: auth get failed: failed to find osd.32 in keyring retval: -2 Module 'cephadm' has failed: auth get failed: failed to find osd.32 in keyring retval: -2 Any idea what i would try to remove that OSD? Thanks, Michael

1 0

mon tried to load "000000.sst" which doesn't exist when recovering from osds
by Yu Wei 29 Jul '20

29 Jul '20

Hi, I deployed rook v0.8.3 with ceph 12.2.7. This is production system deployed for a long time. Because unknown reason, mon couldn't form quorum anymore and I tried to restore mon from osd by following document below, https://github.com/ceph/ceph/blob/v12.2.7/doc/rados/troubleshooting/trouble… After collecting cluster map data, replace store.db and restart mon, monitor log indicated that it tried to load "000000.sst" which was not existed. Log also indicated that mon found all .sst files during startup. Detailed log as below. 2020-07-28 09:44:38.100932 I | rook-ceph-mon2: 2020-07-28 09:44:38.100799 7f2e4abd0ec0 4 rocksdb: CURRENT file: CURRENT 2020-07-28 09:44:38.100946 I | rook-ceph-mon2: 2020-07-28 09:44:38.100951 I | rook-ceph-mon2: 2020-07-28 09:44:38.100847 7f2e4abd0ec0 4 rocksdb: IDENTITY file: IDENTITY 2020-07-28 09:44:38.100958 I | rook-ceph-mon2: 2020-07-28 09:44:38.100963 I | rook-ceph-mon2: 2020-07-28 09:44:38.100865 7f2e4abd0ec0 4 rocksdb: MANIFEST file: MANIFEST-000014 size: 284 Bytes 2020-07-28 09:44:38.100967 I | rook-ceph-mon2: 2020-07-28 09:44:38.100972 I | rook-ceph-mon2: 2020-07-28 09:44:38.100869 7f2e4abd0ec0 4 rocksdb: SST files in /var/lib/rook/rook-ceph-mon2/data/store.db dir, Total Num: 3, files: 000004.sst 000007.sst 000010.sst 2020-07-28 09:44:38.100976 I | rook-ceph-mon2: 2020-07-28 09:44:38.100981 I | rook-ceph-mon2: 2020-07-28 09:44:38.100872 7f2e4abd0ec0 4 rocksdb: Write Ahead Log file in /var/lib/rook/rook-ceph-mon2/data/store.db: 000015.log size: 0 ; 2020-07-28 09:44:38.100985 I | rook-ceph-mon2: 2020-07-28 09:44:38.100989 I | rook-ceph-mon2: 2020-07-28 09:44:38.100874 7f2e4abd0ec0 4 rocksdb: Options.error_if_exists: 0 ... 2020-07-28 09:44:38.101528 I | rook-ceph-mon2: 2020-07-28 09:44:38.101467 7f2e4abd0ec0 4 rocksdb: Fast CRC32 supported: 1 2020-07-28 09:44:38.104667 I | rook-ceph-mon2: 2020-07-28 09:44:38.104317 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000014 2020-07-28 09:44:38.104726 I | rook-ceph-mon2: 2020-07-28 09:44:38.104926 I | rook-ceph-mon2: 2020-07-28 09:44:38.104582 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: ... 2020-07-28 09:44:38.105633 I | rook-ceph-mon2: 2020-07-28 09:44:38.104857 7f2e4abd0ec0 4 rocksdb: Options.report_bg_io_stats: 0 2020-07-28 09:44:38.111205 I | rook-ceph-mon2: 2020-07-28 09:44:38.110905 7f2e4abd0ec0 2 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:1062] Unable to load table properties for file 0 --- IO error: /var/lib/rook/rook-ceph-mon2/data/store.db/000000.sst: No such file or directory 2020-07-28 09:44:38.111266 I | rook-ceph-mon2: 2020-07-28 09:44:38.111693 I | rook-ceph-mon2: 2020-07-28 09:44:38.110999 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/rook/rook-ceph-mon2/data/store.db/MANIFEST-000014 succeeded,manifest_file_number is 14, next_file_number is 17, last_sequence is 111, log_number is 0,prev_log_number is 0,max_column_family is 0 2020-07-28 09:44:38.111723 I | rook-ceph-mon2: 2020-07-28 09:44:38.111732 I | rook-ceph-mon2: 2020-07-28 09:44:38.111006 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 13 2020-07-28 09:44:38.111738 I | rook-ceph-mon2: 2020-07-28 09:44:38.111746 I | rook-ceph-mon2: 2020-07-28 09:44:38.111101 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work 2020-07-28 09:44:38.111764 I | rook-ceph-mon2: 2020-07-28 09:44:38.111214 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl.cc:343] Shutdown complete 2020-07-28 09:44:38.112077 I | rook-ceph-mon2: 2020-07-28 09:44:38.111862 7f2e4abd0ec0 -1 rocksdb: Corruption: Can't access /000000.sst: IO error: /var/lib/rook/rook-ceph-mon2/data/store.db/000000.sst: No such file or directory 2020-07-28 09:44:38.112096 I | rook-ceph-mon2: 2020-07-28 09:44:38.112103 I | rook-ceph-mon2: 2020-07-28 09:44:38.111941 7f2e4abd0ec0 -1 error opening mon data directory at '/var/lib/rook/rook-ceph-mon2/data': (22) Invalid argument 2020-07-28 09:44:38.116813 I | rook-ceph-mon2: 2020-07-28 09:44:38.111862 7f2e4abd0ec0 -1 rocksdb: Corruption: Can't access /000000.sst: IO error: /var/lib/rook/rook-ceph-mon2/data/store.db/000000.sst: No such file or directory 2020-07-28 09:44:38.116874 I | rook-ceph-mon2: 2020-07-28 09:44:38.116883 I | rook-ceph-mon2: 2020-07-28 09:44:38.111941 7f2e4abd0ec0 -1 error opening mon data directory at '/var/lib/rook/rook-ceph-mon2/data': (22) Invalid argument failed to run mon. failed to start mon: Failed to complete 'rook-ceph-mon2': exit status 1. It's a little weird that it tried to load "000000.sst" although rook found the correct three .sst files during startup. There is no "000000.sst" related content in store.db. Any advice for this problem? Is it possible that I executed any wrong steps? Or is there any workaround for this? root@bagwig4:/var/lib/rook/rook-ceph-mon2/data# ls -al store.db/ total 92 drwxr-xr-x 2 root root 188 Jul 28 01:35 . drwxr--r-- 3 root root 55 Jul 28 02:37 .. -rw-r--r-- 1 root root 56547 Jul 28 02:38 000004.sst -rw-r--r-- 1 root root 1179 Jul 28 02:38 000007.sst -rw-r--r-- 1 root root 1243 Jul 28 02:38 000010.sst -rw-r--r-- 1 root root 0 Jul 28 02:38 000015.log -rw-r--r-- 1 root root 16 Jul 28 02:38 CURRENT -rw-r--r-- 1 root root 37 Jul 28 02:38 IDENTITY -rw-r--r-- 1 root root 0 Jul 28 02:38 LOCK -rw-r--r-- 1 root root 284 Jul 28 02:38 MANIFEST-000014 -rw-r--r-- 1 root root 4620 Jul 28 02:38 OPTIONS-000014 -rw-r--r-- 1 root root 4620 Jul 28 02:38 OPTIONS-000017 root@bagwig4:/var/lib/rook/rook-ceph-mon2/data# find . -type f | xargs grep "000000.sst" root@bagwig4:/var/lib/rook/rook-ceph-mon2/data# find . -type f | xargs grep "000000" ./store.db/OPTIONS-000017: delete_obsolete_files_period_micros=21600000000 ./store.db/OPTIONS-000017: memtable_prefix_bloom_size_ratio=0.000000 ./store.db/OPTIONS-000017: max_bytes_for_level_multiplier=10.000000 ./store.db/OPTIONS-000014: delete_obsolete_files_period_micros=21600000000 ./store.db/OPTIONS-000014: memtable_prefix_bloom_size_ratio=0.000000 ./store.db/OPTIONS-000014: max_bytes_for_level_multiplier=10.000000 Thanks, Jared, (韦煜） Software developer Interested in open source software, big data, Linux

1 0

Does Cash app customer service advise me if an account gets checked?
by david william 29 Jul '20

29 Jul '20

At whatever point you are applying to confirm your character on the Cash app, at that point you will likewise get a warning of further advances. All things considered, you have to check your record, on the off chance that you need to go through more cash, or you need to utilize extra services on it. You just need to enter your complete name, date of birth, last four digits of SSN. In the event that there is an issue, contact Cash app customer service.https://www.cash-app-customer-service.com/

1 0

Issue in getting a Cash App refund on account of headway tab? Call help gathering.
by david william 29 Jul '20

29 Jul '20

The advancement tab is one of the basic holds onto the item this is in all probability subject for the Cash App refund. As such, at the off peril which you have to get your coins back, with the guide of using then you may use the help this is given withinside the assistance targets or you may dial the wide variety to interface with the assist group with anticipating any help. https://www.contact-customer-service.net/blog/cash-app-refund/

1 0

Current best practice for migrating from one EC profile to another?
by David Orman 29 Jul '20

29 Jul '20

Hi, As we expand our cluster (adding nodes), we'd like to take advantage of better EC profiles enabled by higher server/rack counts. I understand, as Ceph currently exists (15.2.4), there is no way to live-migrate from one EC profile to another on an existing pool, for example, from 4+2 to 17+3 when going from 7 nodes to 21. Is this correct? How are people accomplishing migrations such as this one (or 4+2 to 9+3, for example) with minimal disruption to services that utilize RBDs sitting on top of these pools? I found: https://ceph.io/geen-categorie/ceph-pool-migration/ , which requires effectively shutting down access during migration (which is doable, but not ideal) - and from what I've read - has some potential downsides (specifically referencing the cppool method). The pools are all EC for data, and an rbd pool exists for metadata. Thank you!

3 2