October 2019 - ceph-users

by Rok Jaklič

Hi, I installed ceph object gateway and I have put one test object onto storage. I can see it with rados -p mytest ls How do I setup ceph that users can access (download,upload) files to this pool? Kind regards, Rok

4 years, 7 months

4
3
0 0

MDS Crashes at “ceph fs volume v011”

by Guilherme Geronimo

Dear ceph users, we're experiencing a segfault during MDS startup (replay process) which is making our FS inaccessible. MDS log messages: Oct 15 03:41:39.894584 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c08f49700 1 -- 192.168.8.195:6800/3181891717 <== osd.26 192.168.8.209:6821/2419345 3 ==== osd_op_reply(21 1.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 154+0+0 (3715233608 0 0) 0x2776340 con 0x18bd500 Oct 15 03:41:39.894584 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched Oct 15 03:41:39.894658 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) _fetched got 0 and 544 Oct 15 03:41:39.894658 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011') Oct 15 03:41:39.894735 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.snaprealm(0x100 seq 1 0x1799c00) open_parents [1,head] Oct 15 03:41:39.894735 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) _fetched [inode 0x100 [...2,head] ~mds0/ auth v275131 snaprealm=0x1799c00 f(v0 1=1+0) n(v76166 rc2020-07-17 15:29:27.000000 b41838692297 -3184=-3168+-16)/n() (iversion lock) 0x18bf800] Oct 15 03:41:39.894821 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched Oct 15 03:41:39.894821 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x1) _fetched got 0 and 482 Oct 15 03:41:39.894891 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x1) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011') Oct 15 03:41:39.894958 mds1 ceph-mds: -472> 2019-10-15 00:40:30.205 7f3c00589700 -1 *** Caught signal (Segmentation fault) **#012 in thread 7f3c00589700 thread_name:fn_anonymous#012#012 ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)#012 1: (()+0x11390) [0x7f3c0e48a390]#012 2: (operator<<(std::ostream&, SnapRealm const&)+0x42) [0x72cb92]#012 3: (SnapRealm::merge_to(SnapRealm*)+0x308) [0x72f488]#012 4: (CInode::decode_snap_blob(ceph::buffer::list&)+0x53) [0x6e1f63]#012 5: (CInode::decode_store(ceph::buffer::list::iterator&)+0x76) [0x702b86]#012 6: (CInode::_fetched(ceph::buffer::list&, ceph::buffer::list&, Context*)+0x1b2) [0x702da2]#012 7: (MDSIOContextBase::complete(int)+0x119) [0x74fcc9]#012 8: (Finisher::finisher_thread_entry()+0x12e) [0x7f3c0ebffece]#012 9: (()+0x76ba) [0x7f3c0e4806ba]#012 10: (clone()+0x6d) [0x7f3c0dca941d]#012 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Oct 15 03:41:39.895400 mds1 ceph-mds: --- logging levels --- Oct 15 03:41:39.895473 mds1 ceph-mds: 0/ 5 none Oct 15 03:41:39.895473 mds1 ceph-mds: 0/ 1 lockdep Cluster status information: cluster: id: b8205875-e56f-4280-9e52-6aab9c758586 health: HEALTH_WARN 1 filesystem is degraded 1 nearfull osd(s) 11 pool(s) nearfull services: mon: 3 daemons, quorum mon1,mon2,mon3 mgr: mon1(active), standbys: mon2, mon3 mds: fs_padrao-1/1/1 up {0=mds1=up:replay(laggy or crashed)} osd: 90 osds: 90 up, 90 in data: pools: 11 pools, 1984 pgs objects: 75.99 M objects, 285 TiB usage: 457 TiB used, 181 TiB / 639 TiB avail pgs: 1896 active+clean 87 active+clean+scrubbing+deep+repair 1 active+clean+scrubbing io: client: 89 KiB/s wr, 0 op/s rd, 3 op/s wr Has anyone seen anything like this? Regards, Arthur

4 years, 7 months

1
0
0 0

CEPH HOLDING : un évènement à organiser ?

by Groupe Partouche

Se désabonner des communications du groupe Partouche L'email ne s'affiche pas correctement ? Cliquez ici. Offre dédiée aux entreprises   Vous souhaitez organiser un séminaire, un cocktail, une soirée de lancement de produit ou tout autre évènement pour votre entreprise ? découvrez nos lieux Offrez-vous des lieux atypiques dans les plus belles destinations de France Nous serons ravis de vous faire bénéficier de 15% de remise* sur votre évènement avec le code OFFREME2019 pour toute nouvelle réservation sur notre site. Confort, détente, divertissement et plaisir des papilles s'allient pour ne faire qu'un, une unité de prestations haut de gamme au service de votre évènement sur-mesure. * 15% de remise pour toute nouvelle réservation entre le 1er novembre 2019 et le 31 mars 2020 sous réserve de disponibilité et pour un minimum de 30 personnes. Ce message vous est adressé par le Groupe Partouche. En application du Règlement du 27 avril 2016 relatif à la protection des données personnelles, vous disposez sur ces données de droits (i) d'accès, (ii) de rectification, (iii) d'effacement, (iv) de limitation et (v) de portabilité de vos données vers un prestataire tiers le cas échéant. Vous pouvez exercer ces droits par courrier postal adressé à : Groupe Partouche / 141 Bis rue de Saussure - 75017 Paris Si vous ne souhaitez plus recevoir d’emails de notre part, vous pouvez vous désinscrire. Conformément aux dispositions de la loi informatique et Libertés du 6 janvier 1978, vous disposez d’un droit d'accès, de rectification et d'opposition. Afin de l'exercer, merci de cliquer sur le lien de désabonnement suivant : Se désabonner Les données de notre Programme proviennent des fichiers des entreprises partenaires et de registres d'entreprises. Vous disposez sur les données personnelles vous concernant de droits d'accès, de rectification, d'effacement, de limitation, d'opposition, de portabilité et du droit de retirer votre consentement à tout moment en suivant ce lien. https://c.evenements-btob.com/unsubscribe?&url=UNSUBSCRIBE&uid=2531…

4 years, 7 months

1
0
0 0

Ceph Day Content & Sponsors Needed

by Mike Perez

Hi everyone, We have some exciting Ceph Days coming up and we're still looking for content and sponsors. Take a look: * Ceph Day Argentina (Spanish speakers only) - October 16: https://ceph.io/cephdays/ceph-day-argentina-2019/ * Ceph Day London - October 24: https://ceph.io/cephdays/ceph-day-london-2019/ * Ceph Day Poland - October 28: https://ceph.io/cephdays/ceph-day-poland-2019/ * Ceph Day San Diego - November 18: https://ceph.io/cephdays/ceph-rook-day-san-diego-2019/ If you or you know someone who can speak at these events please follow the call for proposal links on each page. If you need help working on your title/abstract, please feel free to reach out to me directly. If your company is interested in sponsoring a Ceph Day, please take a look at our brochure: https://ceph.io/wp-content/uploads/2019/10/Ceph-Day-Partner-Sponsorship.pdf Thanks! -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

4 years, 7 months

1
1
0 0

MDS Crashes on “ceph fs volume v011”

by Guilherme Geronimo

Dear ceph users, we're experiencing a segfault during MDS startup (replay process) which is making our FS inaccessible. MDS log messages: Oct 15 03:41:39.894584 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c08f49700 1 -- 192.168.8.195:6800/3181891717 <== osd.26 192.168.8.209:6821/2419345 3 ==== osd_op_reply(21 1.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 154+0+0 (3715233608 0 0) 0x2776340 con 0x18bd500 Oct 15 03:41:39.894584 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched Oct 15 03:41:39.894658 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) _fetched got 0 and 544 Oct 15 03:41:39.894658 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011') Oct 15 03:41:39.894735 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.snaprealm(0x100 seq 1 0x1799c00) open_parents [1,head] Oct 15 03:41:39.894735 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) _fetched [inode 0x100 [...2,head] ~mds0/ auth v275131 snaprealm=0x1799c00 f(v0 1=1+0) n(v76166 rc2020-07-17 15:29:27.000000 b41838692297 -3184=-3168+-16)/n() (iversion lock) 0x18bf800] Oct 15 03:41:39.894821 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched Oct 15 03:41:39.894821 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x1) _fetched got 0 and 482 Oct 15 03:41:39.894891 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x1) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011') Oct 15 03:41:39.894958 mds1 ceph-mds: -472> 2019-10-15 00:40:30.205 7f3c00589700 -1 *** Caught signal (Segmentation fault) **#012 in thread 7f3c00589700 thread_name:fn_anonymous#012#012 ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)#012 1: (()+0x11390) [0x7f3c0e48a390]#012 2: (operator<<(std::ostream&, SnapRealm const&)+0x42) [0x72cb92]#012 3: (SnapRealm::merge_to(SnapRealm*)+0x308) [0x72f488]#012 4: (CInode::decode_snap_blob(ceph::buffer::list&)+0x53) [0x6e1f63]#012 5: (CInode::decode_store(ceph::buffer::list::iterator&)+0x76) [0x702b86]#012 6: (CInode::_fetched(ceph::buffer::list&, ceph::buffer::list&, Context*)+0x1b2) [0x702da2]#012 7: (MDSIOContextBase::complete(int)+0x119) [0x74fcc9]#012 8: (Finisher::finisher_thread_entry()+0x12e) [0x7f3c0ebffece]#012 9: (()+0x76ba) [0x7f3c0e4806ba]#012 10: (clone()+0x6d) [0x7f3c0dca941d]#012 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Oct 15 03:41:39.895400 mds1 ceph-mds: --- logging levels --- Oct 15 03:41:39.895473 mds1 ceph-mds: 0/ 5 none Oct 15 03:41:39.895473 mds1 ceph-mds: 0/ 1 lockdep Cluster status information: cluster: id: b8205875-e56f-4280-9e52-6aab9c758586 health: HEALTH_WARN 1 filesystem is degraded 1 nearfull osd(s) 11 pool(s) nearfull services: mon: 3 daemons, quorum mon1,mon2,mon3 mgr: mon1(active), standbys: mon2, mon3 mds: fs_padrao-1/1/1 up {0=mds1=up:replay(laggy or crashed)} osd: 90 osds: 90 up, 90 in data: pools: 11 pools, 1984 pgs objects: 75.99 M objects, 285 TiB usage: 457 TiB used, 181 TiB / 639 TiB avail pgs: 1896 active+clean 87 active+clean+scrubbing+deep+repair 1 active+clean+scrubbing io: client: 89 KiB/s wr, 0 op/s rd, 3 op/s wr Has anyone seen anything like this? Regards, []'s Arthur Enviado via iPhone

4 years, 7 months

1
0
0 0

Crashed MDS (segfault)

by Gustavo Tonini

Dear ceph users, we're experiencing a segfault during MDS startup (replay process) which is making our FS inaccessible. MDS log messages: Oct 15 03:41:39.894584 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c08f49700 1 -- 192.168.8.195:6800/3181891717 <== osd.26 192.168.8.209:6821/2419345 3 ==== osd_op_reply(21 1.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 154+0+0 (3715233608 0 0) 0x2776340 con 0x18bd500 Oct 15 03:41:39.894584 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched Oct 15 03:41:39.894658 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) _fetched got 0 and 544 Oct 15 03:41:39.894658 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011') Oct 15 03:41:39.894735 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.snaprealm(0x100 seq 1 0x1799c00) open_parents [1,head] Oct 15 03:41:39.894735 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) _fetched [inode 0x100 [...2,head] ~mds0/ auth v275131 snaprealm=0x1799c00 f(v0 1=1+0) n(v76166 rc2020-07-17 15:29:27.000000 b41838692297 -3184=-3168+-16)/n() (iversion lock) 0x18bf800] Oct 15 03:41:39.894821 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched Oct 15 03:41:39.894821 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x1) _fetched got 0 and 482 Oct 15 03:41:39.894891 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x1) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011') Oct 15 03:41:39.894958 mds1 ceph-mds: -472> 2019-10-15 00:40:30.205 7f3c00589700 -1 *** Caught signal (Segmentation fault) **#012 in thread 7f3c00589700 thread_name:fn_anonymous#012#012 ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)#012 1: (()+0x11390) [0x7f3c0e48a390]#012 2: (operator<<(std::ostream&, SnapRealm const&)+0x42) [0x72cb92]#012 3: (SnapRealm::merge_to(SnapRealm*)+0x308) [0x72f488]#012 4: (CInode::decode_snap_blob(ceph::buffer::list&)+0x53) [0x6e1f63]#012 5: (CInode::decode_store(ceph::buffer::list::iterator&)+0x76) [0x702b86]#012 6: (CInode::_fetched(ceph::buffer::list&, ceph::buffer::list&, Context*)+0x1b2) [0x702da2]#012 7: (MDSIOContextBase::complete(int)+0x119) [0x74fcc9]#012 8: (Finisher::finisher_thread_entry()+0x12e) [0x7f3c0ebffece]#012 9: (()+0x76ba) [0x7f3c0e4806ba]#012 10: (clone()+0x6d) [0x7f3c0dca941d]#012 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Oct 15 03:41:39.895400 mds1 ceph-mds: --- logging levels --- Oct 15 03:41:39.895473 mds1 ceph-mds: 0/ 5 none Oct 15 03:41:39.895473 mds1 ceph-mds: 0/ 1 lockdep Cluster status information: cluster: id: b8205875-e56f-4280-9e52-6aab9c758586 health: HEALTH_WARN 1 filesystem is degraded 1 nearfull osd(s) 11 pool(s) nearfull services: mon: 3 daemons, quorum mon1,mon2,mon3 mgr: mon1(active), standbys: mon2, mon3 mds: fs_padrao-1/1/1 up {0=mds1=up:replay(laggy or crashed)} osd: 90 osds: 90 up, 90 in data: pools: 11 pools, 1984 pgs objects: 75.99 M objects, 285 TiB usage: 457 TiB used, 181 TiB / 639 TiB avail pgs: 1896 active+clean 87 active+clean+scrubbing+deep+repair 1 active+clean+scrubbing io: client: 89 KiB/s wr, 0 op/s rd, 3 op/s wr Has anyone seen anything like this? Regards, Gustavo.

4 years, 7 months

1
0
0 0

run-s3tests.sh against Nautilus

by Francisco Londono

I ran the s3 test (run-s3tests.sh) in vstart mode against Nautilus. Is there a better guide out there than this one? https://docs.ceph.com/ceph-prs/17381/dev/#testing-how-to-run-s3-tests-local… Thus far I ran into a ceph.conf parse issue, a keyring permission issue and a radosgw crash. + radosgw-admin user create --uid=s3test1 --display-name=tester1 --access-key=access1 --secret=secret1 --email=tester1(a)ceph.com server name not found: [v2:xx.x.x.xx:40466 (Name or service not known) unable to parse addrs in '[v2:xx.x.x.xx:40466,v1:xx.x.x.xx:40467]' couldn't init storage provider The ceph.conf under the global header shows: mon host = [v2:xx.x.x.xx:40466,v1:xx.x.x.xx:40467] In Luminous, the ceph.conf created used to be in this format: host = (hostname) mon addr = xx.x.x.xx:6789 Thanks.

4 years, 7 months

1
0
0 0

Librados in openstack

by solarflow99

I was wondering if this is provided somehow? All I see is rbd and radosgw mentioned. If you have applications built with librados surely openstack must have a way to provide it?

4 years, 7 months

1
0
0 0

Recurring issue: PG is inconsistent, but lists no inconsistent objects

by Florian Haas

Hello, I am running into an "interesting" issue with a PG that is being flagged as inconsistent during scrub (causing the cluster to go to HEALTH_ERR), but doesn't actually appear to contain any inconsistent objects. $ ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent OSD_SCRUB_ERRORS 1 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 10.10d is active+clean+inconsistent, acting [15,13] $ rados list-inconsistent-obj 10.10d {"epoch":12138,"inconsistents":[]} "ceph pg query" (see below) on that PG does report num_scrub_errors=1, num_shallow_scrub_errors=1, and num_objects_dirty=1. "osd scrub auto repair = true" is set on all OSDs, but the PG never auto-repairs. (This is a test cluster, the pool size is 2 — this may preclude auto repair from ever kicking in; I'm not sure on that one.) "ceph pg repair" does repair, but the issue reappears on the next scheduled scrub. This issue was first discovered while the cluster was on Jewel/Filestore. In an event like this I would normally suspect either a problem with an individual OSD, or a bug in the FileStore code. But the cluster has had *all* of it's OSDs replaced since, as part of a full Jewel→Luminous→Nautilus upgrade and a FileStore→BlueStore conversion. The issue still persists. A full "ceph pg 10.10d query" result is below. If anyone has ideas on how to permanently fix this issue, I'd be most grateful. Thanks! Cheers, Florian { "state": "active+clean+inconsistent", "snap_trimq": "[]", "snap_trimq_len": 0, "epoch": 12143, "up": [ 15, 13 ], "acting": [ 15, 13 ], "acting_recovery_backfill": [ "13", "15" ], "info": { "pgid": "10.10d", "last_update": "100'11", "last_complete": "100'11", "log_tail": "0'0", "last_user_version": 11, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [], "history": { "epoch_created": 45, "epoch_pool_created": 45, "last_epoch_started": 12139, "last_interval_started": 12138, "last_epoch_clean": 12139, "last_interval_clean": 12138, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 12138, "same_interval_since": 12138, "same_primary_since": 12114, "last_scrub": "100'11", "last_scrub_stamp": "2019-10-14 08:33:57.347097", "last_deep_scrub": "100'11", "last_deep_scrub_stamp": "2019-10-11 14:09:29.016946", "last_clean_scrub_stamp": "2019-10-11 14:09:29.016946" }, "stats": { "version": "100'11", "reported_seq": "4927", "reported_epoch": "12143", "state": "active+clean+inconsistent", "last_fresh": "2019-10-14 08:33:57.347147", "last_change": "2019-10-14 08:33:57.347147", "last_active": "2019-10-14 08:33:57.347147", "last_peered": "2019-10-14 08:33:57.347147", "last_clean": "2019-10-14 08:33:57.347147", "last_became_active": "2019-10-11 14:44:09.312226", "last_became_peered": "2019-10-11 14:44:09.312226", "last_unstale": "2019-10-14 08:33:57.347147", "last_undegraded": "2019-10-14 08:33:57.347147", "last_fullsized": "2019-10-14 08:33:57.347147", "mapping_epoch": 12138, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 45, "last_epoch_clean": 12139, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "100'11", "last_scrub_stamp": "2019-10-14 08:33:57.347097", "last_deep_scrub": "100'11", "last_deep_scrub_stamp": "2019-10-11 14:09:29.016946", "last_clean_scrub_stamp": "2019-10-11 14:09:29.016946", "log_size": 11, "ondisk_log_size": 11, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": true, "pin_stats_invalid": true, "manifest_stats_invalid": true, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 11, "num_objects": 1, "num_object_clones": 0, "num_object_copies": 2, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 1, "num_whiteouts": 0, "num_read": 33, "num_read_kb": 22, "num_write": 11, "num_write_kb": 6, "num_scrub_errors": 1, "num_shallow_scrub_errors": 1, "num_deep_scrub_errors": 0, "num_objects_recovered": 2, "num_bytes_recovered": 22, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 15, 13 ], "acting": [ 15, 13 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 15, "acting_primary": 15, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 12139, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [ { "peer": "13", "pgid": "10.10d", "last_update": "100'11", "last_complete": "100'11", "log_tail": "0'0", "last_user_version": 11, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [], "history": { "epoch_created": 45, "epoch_pool_created": 45, "last_epoch_started": 12139, "last_interval_started": 12138, "last_epoch_clean": 12139, "last_interval_clean": 12138, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 12138, "same_interval_since": 12138, "same_primary_since": 12114, "last_scrub": "100'11", "last_scrub_stamp": "2019-10-14 08:33:57.347097", "last_deep_scrub": "100'11", "last_deep_scrub_stamp": "2019-10-11 14:09:29.016946", "last_clean_scrub_stamp": "2019-10-11 14:09:29.016946" }, "stats": { "version": "100'11", "reported_seq": "36", "reported_epoch": "12113", "state": "active+undersized+degraded", "last_fresh": "2019-10-11 14:39:58.946532", "last_change": "2019-10-11 14:39:58.924989", "last_active": "2019-10-11 14:39:58.946532", "last_peered": "2019-10-11 14:39:58.946532", "last_clean": "2014-11-05 15:48:35.131248", "last_became_active": "2019-10-11 14:39:58.924989", "last_became_peered": "2019-10-11 14:39:58.924989", "last_unstale": "2019-10-11 14:39:58.946532", "last_undegraded": "2019-10-11 14:39:58.892352", "last_fullsized": "2019-10-11 14:39:58.892331", "mapping_epoch": 12138, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 45, "last_epoch_clean": 12103, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "100'11", "last_scrub_stamp": "2019-10-11 14:09:29.016946", "last_deep_scrub": "100'11", "last_deep_scrub_stamp": "2019-10-11 14:09:29.016946", "last_clean_scrub_stamp": "2019-10-11 14:09:29.016946", "log_size": 11, "ondisk_log_size": 11, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": true, "pin_stats_invalid": true, "manifest_stats_invalid": true, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 11, "num_objects": 1, "num_object_clones": 0, "num_object_copies": 2, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 1, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 1, "num_whiteouts": 0, "num_read": 33, "num_read_kb": 22, "num_write": 11, "num_write_kb": 6, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 2, "num_bytes_recovered": 22, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 15, 13 ], "acting": [ 15, 13 ], "avail_no_missing": [ "13" ], "object_location_counts": [ { "shards": "13", "objects": 1 } ], "blocked_by": [], "up_primary": 15, "acting_primary": 15, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 12139, "hit_set_history": { "current_last_update": "0'0", "history": [] } } ], "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2019-10-11 14:44:09.175574", "might_have_unfound": [], "recovery_progress": { "backfill_targets": [], "waiting_on_backfill": [], "last_backfill_started": "MIN", "backfill_info": { "begin": "MIN", "end": "MIN", "objects": [] }, "peer_backfill_info": [], "backfills_in_flight": [], "recovering": [], "pg_backend": { "pull_from_peer": [], "pushing": [] } }, "scrub": { "scrubber.epoch_start": "12138", "scrubber.active": false, "scrubber.state": "INACTIVE", "scrubber.start": "MIN", "scrubber.end": "MIN", "scrubber.max_end": "MIN", "scrubber.subset_last_update": "0'0", "scrubber.deep": false, "scrubber.waiting_on_whom": [] } }, { "name": "Started", "enter_time": "2019-10-11 14:44:08.833757" } ], "agent_state": {} }

4 years, 7 months

3
8
0 0

Corrupted block.db for osd. How to extract particular PG from that osd?

by Alexey Kalinkin

Hello cephers! We're lost block.db file for one of our osd. This results in down osd and incomplete PG. Block file from osd, which symlinks to particular /dev folder is live and correct. My main question is there any thereoretical possibility to extract particular PG from osd which block.db device has been lost? Is there any way I can make it using ceph-objectstore-tool or ceph-bluestore-tool? We're running ceph version 12.2.12 -- Very truly yours, Kalinkin Alexey Research fellow of Epigenetic lab, RCMG.

4 years, 7 months

2
1
0 0

2024

2023

2022

2021

2020

2019

ceph-users October 2019