- ceph-users - lists.ceph.io

RBD pool damaged, repair options?
by Robert Sander 13 Aug '20

13 Aug '20

Hi, a customer lost 5 OSDs at the same time (and replaced them with new disks before we could do anything…). 4 PGs were incomplete but could be repaired with ceph-objectstore-tool. The cluster itself is healthy again. Now some RBDs are missing. They are still listed in the rbd_directory object but cannot be access with e.g. rbd info. rbd ls says: no such file or directory I assume that is because the associated rbd_id.name object has been lost. I tried to create this object by means of "rados put" but still no luck: 2020-08-13 11:47:27.557 7fa1e2ffd700 -1 librbd::image::OpenRequest: failed to retrieve image id: (5) Input/output error 2020-08-13 11:47:27.557 7fa1e27fc700 -1 librbd::ImageState: 0x55b9ebb75b10 failed to open image: (5) Input/output error 2020-08-13 11:47:27.557 7fa1e27fc700 -1 librbd::io::AioCompletion: 0x55b9ebadd3a0 fail: (5) Input/output error rbd: error opening vm-501-disk-2: (5) Input/output error What are the options to repair an RBD pool so that at least the RBDs (most data objects are still there) are available again. Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

Approach Epson Customer Service For Fixing Network Problems.
by mary smith 13 Aug '20

13 Aug '20

Yes, you are not only allowed to approach Epson Customer Service but also fetch the instant technical backing directly from the professionals. If you want to avail the quickest remedy along with the supervision regarding the problems and hurdles, pertaining to HP printer network, you should get the free assistance at anytime. https://www.epsonprintersupportpro.net/

1 0

Block Network IP Of Wireless HP Printer Using HP Support Assistant.
by mary smith 13 Aug '20

13 Aug '20

Yes, you should! If you are one of those who are have a word with the professionals who are capable of resolving the whole host of network problems. You can make use of HP Support Assistant who will solve the whole host of wireless printer problems. https://www.amiytech.com/hp-support-assistant/

1 0

Heavy rocksdb activity in newly added osd
by Jerry Pu 13 Aug '20

13 Aug '20

Hi All We had a cluster (v13.2.4) with 32 osds in total. At first, an osd (osd.18) in cluster was down. So, we tried to remove the osd and added a new one (osd.32) with new ID. We unplugged the disk (osd.18) and plugged in a new disk in the same slot and add osd.32 into cluster. Then, osd.32 was booting, but, we found it takes much time (around 18 mins) for the osd to change to up state. Diving into osd.32 logs, we see that there is much rocksdb activity before osd.32 change to up state. Can anyone explain why this happened or give me any advice about how to prevent from this. Thanks. [osd.32 log] 2020-08-03 15:36:58.852 7f88021fa1c0 0 osd.32 0 done with init, starting boot process 2020-08-03 15:36:58.852 7f88021fa1c0 1 osd.32 0 start_boot 2020-08-03 15:36:58.854 7f87db02b700 -1 osd.32 0 waiting for initial osdmap 2020-08-03 15:36:58.855 7f87e4ba0700 -1 osd.32 0 failed to load OSD map for epoch 22010, got 0 bytes 2020-08-03 15:36:58.955 7f87e0836700 0 osd.32 22011 crush map has features 283675107524608, adjusting msgr requires for clients 2020-08-03 15:36:58.955 7f87e0836700 0 osd.32 22011 crush map has features 283675107524608 was 288232575208792577, adjusting msgr requires for mons *2020-08-03 15:36:58.955* 7f87e0836700 0 osd.32 22011 crush map has features 720859615486820352, adjusting msgr requires for osds 2020-08-03 15:37:31.182 7f87e1037700 4 rocksdb: [/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/db_impl_write.cc:1346] [default] New memtable created with log file: #16. Immutable memtables: 0. 2020-08-03 15:37:31.285 7f87e8045700 4 rocksdb: (Original Log Time 2020/08/03-15:37:31.183995) [/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/db_impl_compaction_flush.cc:1396] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2020-08-03 15:37:31.285 7f87e8045700 4 rocksdb: [/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/flush_job.cc:300] [default] [JOB 3] Flushing memtable with next log file: 16 -------- lots of rocksdb activity--------- 2020-08-03 15:54:21.704 7f87e8045700 4 rocksdb: (Original Log Time 2020/08/03-15:54:21.705680) [/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/memtable_list.cc:397] [default] Level-0 commit table #112: memtable #1 done 2020-08-03 15:54:21.704 7f87e8045700 4 rocksdb: (Original Log Time 2020/08/03-15:54:21.705704) EVENT_LOG_v1 {"time_micros": 1596441261705697, "job": 51, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 3, 0, 0, 0, 0, 0], "immutable_memtables": 0} 2020-08-03 15:54:21.704 7f87e8045700 4 rocksdb: (Original Log Time 2020/08/03-15:54:21.705721) [/home/gitlab/rpmbuild/BUILD/ceph-13.2.4/src/rocksdb/db/db_impl_compaction_flush.cc:172] [default] Level summary: base level 1 max bytes base 268435456 files[1 3 0 0 0 0 0] max score 0.75 *2020-08-03 15:54:38.567* 7f87e0836700 1 osd.32 502096 state: booting -> active 2020-08-03 15:54:38.567 7f87d5820700 1 osd.32 pg_epoch: 502096 pg[1.17e( empty local-lis/les=0/0 n=0 ec=11627/16 lis/c 501703/501703 les/c/f 501704/501704/0 502096/502096/502096) [32,26,28] r=0 lpr=502096 pi=[501703,502096)/1 crt=0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary Best Jerry

1 0

Can I install HP envy 4520 drivers correctly?
by Vin Diesel 13 Aug '20

13 Aug '20

HP envy 4520 wireless printer is one of the most popular brands, which is specially known for its printing services. It is the best product in the features and functions, so millions of users prefer it for the best printing performance. Due to some technical problems, my HP 4520 drivers are fully corrupted, so I want to install the newest version of HP envy 4520 drivers successfully. As per my technical experience, I am trying my skills to install HP envy 4520 printer driver. I am failing to complete installation process. So anyone can help me to install printer driver appropriately. https://www.hpprintersupportpro.com/blog/troubleshoot-hp-printer-envy-4520-…

1 0

Can I install HP envy 4520 drivers correctly?
by Vin Diesel 13 Aug '20

13 Aug '20

HP envy 4520 wireless printer is one of the most popular brands, which is specially known for its printing services. It is the best product in the features and functions, so millions of users prefer it for the best printing performance. Due to some technical problems, my HP 4520 drivers are fully corrupted, so I want to install the newest version of [url=https://www.hpprintersupportpro.com/blog/troubleshoot-hp-printer-envy-4… envy 4520 drivers[/url] successfully. As per my technical experience, I am trying my skills to install HP envy 4520 printer driver. I am failing to complete installation process. So anyone can help me to install printer driver appropriately.

1 0

Ceph not warning about clock skew on an OSD-only host?
by Matthew Vernon 13 Aug '20

13 Aug '20

Hi, Our production cluster runs Luminous. Yesterday, one of our OSD-only hosts came up with its clock about 8 hours wrong(!) having been out of the cluster for a week or so. Initially, ceph seemed entirely happy, and then after an hour or so it all went South (OSDs start logging about bad authenticators, I/O pauses, general sadness). I know clock sync is important to Ceph, so "one system is 8 hours out, Ceph becomes sad" is not a surprise. It is perhaps a surprise that the OSDs were allowed in at all... What _is_ a surprise, though, is that at no point in all this did Ceph raise a peep about clock skew. Normally it's pretty sensitive to this - our test cluster has had clock skew complaints when a mon is only slightly out, and here we had a node 8 hours wrong. Is there some oddity like Ceph not warning on clock skew for OSD-only hosts? or an upper bound on how high a discrepency it will WARN about? Regards, Matthew example output from mid-outage: root@sto-3-1:~# ceph -s cluster: id: 049fc780-8998-45a8-be12-d3b8b6f30e69 health: HEALTH_ERR 40755436/2702185683 objects misplaced (1.508%) Reduced data availability: 20 pgs inactive, 20 pgs peering Degraded data redundancy: 367431/2702185683 objects degraded (0.014%), 4549 pgs degraded 481 slow requests are blocked > 32 sec. Implicated osds 188,284,795,1278,1981,2061,2648,2697 644 stuck requests are blocked > 4096 sec. Implicated osds 22,31,33,35,101,116,120,130,132,140,150,159,201,211,228,263,327,541,561,566,585,589,636,643,649,654,743,785,790,806,865,1037,1040,1090,1100,1104,1115,1134,1135,1166,1193,1275,1277,1292,1494,1523,1598,1638,1746,2055,2069,2191,2210,2358,2399,2486,2487,2562,2589,2613,2627,2656,2713,2720,2837,2839,2863,2888,2908,2920,2928,2929,2947,2948,2963,2969,2972 [...] -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

3 2

fio rados ioengine
by Frank Ritchie 12 Aug '20

12 Aug '20

Hi all, Does anyone know if using fio with the rados ioengine will produce similar results to rados bench, i.e. can the 2 be used interchangeably? thx Frank

1 0

RGW unable to delete a bucket
by Andrei Mikhailovsky 12 Aug '20

12 Aug '20

Hi I am trying to delete a bucket using the following command: # radosgw-admin bucket rm --bucket=<bucket-name> --purge-objects However, in console I get the following messages. About 100+ of those messages per second. 2020-08-04T17:11:06.411+0100 7fe64cacf080 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 The command has been running for about 35 days days and it still hasn't finished. The size of the bucket is under 1TB for sure. Probably around 500GB. I have recently removed about a dozen of old buckets without any issues. It's this particular bucket that is being very stubborn. Anything I can do to remove it, including it's objects and any orphans it might have? Thanks Andrei

3 3

DocuBetter Meeting Today 1630 UTC
by John Zachary Dover 12 Aug '20

12 Aug '20

There is a general documentation meeting called the "DocuBetter Meeting", and it is held every two weeks. The next DocuBetter Meeting will be on 12 Aug 2020 at 0830 PDT, and will run for thirty minutes. Everyone with a documentation-related request or complaint is invited. The meeting will be held here: https://bluejeans.com/908675367 This meeting will cover the reorganization of the Ceph website as well as Zac's recently-developed workflow that aims to make it possible to move the good ideas from stale or poorly-formed documentation PRs into the documentation more quickly. Send documentation-related requests and complaints to me by replying to this email and CCing me at zac.dover(a)gmail.com. The next DocuBetter meeting is scheduled for: 12 Aug 2020 0830 PDT 12 Aug 2020 1630 UTC 13 Aug 2020 0230 AEST Etherpad: https://pad.ceph.com/p/Ceph_Documentation Zac's docs whiteboard: https://pad.ceph.com/p/docs_whiteboard Meeting: https://bluejeans.com/908675367

1 0