- ceph-users - lists.ceph.io

Cephalocon 2020 will be March 4-5 in Seoul, South Korea!
by Sage Weil 20 Nov '19

20 Nov '19

Hi everyone, We're pleased to announce that the next Cephalocon will be March 3-5 in Seoul, South Korea! https://ceph.com/cephalocon/seoul-2020/ The CFP for the conference is now open: https://linuxfoundation.smapply.io/prog/cephalocon_2020 Main conference: March 4-5 Developer summit: March 3 Mark your calendars, and get your talk proposals in! The CFP will close in early December in order to get a final schedule published in early January. In addition to the two day conference, we will also have a developer summit on March 3 to take advantage of having so many developers in the same place at the same time. The developer sessions will include video conferencing so that remote developers will also be able to participate. A sponsorship prospectus will be available Real Soon Now. We hope you can join us!

1 0

scrub error on object storage pool
by M Ranga Swami Reddy 20 Nov '19

20 Nov '19

Hello - Recently we have upgraded to Luminous 12.2.11. After that we can see the scrub errors on the object storage pool only on daily basis. After repair, it will be cleared. But again it will come tomorrow after scrub performed the PG. Any known issue - on scrub errs with 12.2.11 version? Thanks Swami

1 0

msgr2 not used on OSDs in some Nautilus clusters
by Bryan Stillwell 19 Nov '19

19 Nov '19

I've upgraded 7 of our clusters to Nautilus (14.2.4) and noticed that on some of the clusters (3 out of 7) the OSDs aren't using msgr2 at all. Here's the output for osd.0 on 2 clusters of each type: ### Cluster 1 (v1 only): # ceph osd find 0 | jq -r '.addrs' { "addrvec": [ { "type": "v1", "addr": "10.26.0.33:6809", "nonce": 4185021 } ] } ### Cluster 2 (v1 only): # ceph osd find 0 | jq -r '.addrs' { "addrvec": [ { "type": "v1", "addr": "10.197.0.243:6801", "nonce": 3802140 } ] } ### Cluster 3 (v1 & v2): # ceph osd find 0 | jq -r '.addrs' { "addrvec": [ { "type": "v2", "addr": "10.32.0.36:6802", "nonce": 3167 }, { "type": "v1", "addr": "10.32.0.36:6804", "nonce": 3167 } ] } ### Cluster 4 (v1 & v2): # ceph osd find 0 | jq -r '.addrs' { "addrvec": [ { "type": "v2", "addr": "10.36.0.12:6820", "nonce": 3150 }, { "type": "v1", "addr": "10.36.0.12:6827", "nonce": 3150 } ] } All of the mon nodes have the same msgr settings of: # ceph daemon mon.$(hostname -s) config show | grep msgr "mon_warn_on_msgr2_not_enabled": "true", "ms_bind_msgr1": "true", "ms_bind_msgr2": "true", "ms_msgr2_encrypt_messages": "false", "ms_msgr2_sign_messages": "false", What could be causing this? All of the clusters are listening on port 3300 for v2 and 6789 for v1. I can even connect to port 3300 on the mon nodes from the OSD nodes. Thanks, Bryan

2 4

jewel OSDs refuse to start up again
by Janne Johansson 19 Nov '19

19 Nov '19

Three OSDs, holding the 3 replicas of a PG here are only half-starting, and hence that single PG gets stuck as "stale+active+clean". All died of suicide timeout while walking over a huge omap (pool 7 'default.rgw.buckets.index') and would not get the PG 7.b back online again. From the logs, they try to start normally, get into a bit of leveldb things, play the journal and then say nothing more. 2019-11-19 15:15:46.967543 7fe644fad840 0 set uid:gid to 167:167 (ceph:ceph) 2019-11-19 15:15:46.967600 7fe644fad840 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-osd, pid 5149 2019-11-19 15:15:47.026065 7fe644fad840 0 pidfile_write: ignore empty --pid-file 2019-11-19 15:15:47.078291 7fe644fad840 0 filestore(/var/lib/ceph/osd/ceph-22) backend xfs (magic 0x58465342) 2019-11-19 15:15:47.079317 7fe644fad840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-22) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2019-11-19 15:15:47.079331 7fe644fad840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-22) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option 2019-11-19 15:15:47.079352 7fe644fad840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-22) detect_features: splice is supported 2019-11-19 15:15:47.080287 7fe644fad840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-22) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2019-11-19 15:15:47.080529 7fe644fad840 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-22) detect_feature: extsize is disabled by conf 2019-11-19 15:15:47.095819 7fe644fad840 1 leveldb: Recovering log #2731809 2019-11-19 15:15:47.119792 7fe644fad840 1 leveldb: Level-0 table #2731812: started 2019-11-19 15:15:47.132107 7fe644fad840 1 leveldb: Level-0 table #2731812: 140642 bytes OK 2019-11-19 15:15:47.143782 7fe644fad840 1 leveldb: Delete type=0 #2731809 2019-11-19 15:15:47.147198 7fe644fad840 1 leveldb: Delete type=3 #2731792 2019-11-19 15:15:47.159339 7fe644fad840 0 filestore(/var/lib/ceph/osd/ceph-22) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2019-11-19 15:15:47.243262 7fe644fad840 1 journal _open /var/lib/ceph/osd/ceph-22/journal fd 18: 21472739328 bytes, block size 4096 bytes, directio = 1, aio = 1 At this point they consume a ton of cpu, systemd thinks all is fine, and this has been going on for some 5 hours. ceph -s think they are down, I can't talk to the OSDs remotely from a mon, but ceph daemon on the OSD hosts works normally, except I can't do anything from there except get conf or perf numbers. Strace shows they all keep looping over the same sequence: machine1: stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7fffd7c98080) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 stat("/var/lib/ceph/osd/ceph-270/current/7.b_head", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7fffd7c98080) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 machine2: stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7ffe0b664240) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 stat("/var/lib/ceph/osd/ceph-243/current/7.b_head", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7ffe0b664240) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 machine3: stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7ffc63518650) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 stat("/var/lib/ceph/osd/ceph-22/current/7.b_head", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7ffc63518650) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 Help wanted. -- May the most significant bit of your life be positive.

1 0

How proceed to change a crush rule and remap pg's?
by Maarten van Ingen 19 Nov '19

19 Nov '19

Hi, I have a small but impacting error in my crush rules. For unknown reasons the rules are not using host but osd to place the data and thus we have some nodes with all three copies instead of three different nodes. We noticed this when rebooting a node and a pg became stale. My crush rule: { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "osd" }, { "op": "emit" } ] }, Type should be host of course. And I want to alter this and move pg's such that all is as should. How can I best proceed in correcting this issue? I do like to throttle the remapping of the data so ceph itself won't be unavailable while the data is redistributed. We are running on Mimic (13.2.6), and this environment has been installed freshly as Mimic while using ceph-ansible. Current ceph -s output: cluster: id: <<fsid> health: HEALTH_OK services: mon: 3 daemons, quorum mon01,mon02,mon03 mgr: mon01(active), standbys: mon02, mon03 mds: cephfs-2/2/2 up {0=mon03=up:active,1=mon01=up:active}, 1 up:standby osd: 502 osds: 502 up, 502 in data: pools: 18 pools, 8192 pgs objects: 28.74 M objects, 100 TiB usage: 331 TiB used, 2.3 PiB / 2.6 PiB avail pgs: 8192 active+clean Cheers, Maarten van Ingen | Systems Expert | Distributed Data Processing | SURFsara | Science Park 140 | 1098 XG Amsterdam | | T +31 (0) 20 800 1300 | maarten.vaningen(a)surfsara.nl | https://surfsara.nl | We are ISO 27001 certified and meet the high requirements for information security.

2 2

corporate video production company in bangalore
by vhtnow11＠gmail.com 19 Nov '19

19 Nov '19

Best Video Production Company in Bangalore : We at VHTnow create visual masterpieces that engage, inspire and impact people's lives. Our services also include ad film and corporate film production in bangalore visit:https://vhtnow.com/

1 0

Re: Ssd cache question
by Wesley Peng 19 Nov '19

19 Nov '19

1 0

Ssd cache question
by Wesley Peng 18 Nov '19

18 Nov '19

2 1

Ceph manager causing MGR active switch
by Thomas Schneider 18 Nov '19

18 Nov '19

Hi, I can see the following error message regularely in MGR log: 2019-11-18 14:25:48.847 7fd9e6a3a700 0 mgr[dashboard] [18/Nov/2019:14:25:48] ENGINE Error in HTTPServer.tick Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2021, in start self.tick() File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2090, in tick s, ssl_env = self.ssl_adapter.wrap(s) File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/ssl_builtin.py", line 67, in wrap server_side=True) File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket _context=self) File "/usr/lib/python2.7/ssl.py", line 599, in __init__ self.do_handshake() File "/usr/lib/python2.7/ssl.py", line 828, in do_handshake self._sslobj.do_handshake() error: [Errno 0] Error 2019-11-18 14:25:49.027 7fd9e6a3a700 0 mgr[dashboard] [18/Nov/2019:14:25:49] ENGINE Error in HTTPServer.tick Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2021, in start self.tick() File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2090, in tick s, ssl_env = self.ssl_adapter.wrap(s) File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/ssl_builtin.py", line 67, in wrap server_side=True) File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket _context=self) File "/usr/lib/python2.7/ssl.py", line 599, in __init__ self.do_handshake() File "/usr/lib/python2.7/ssl.py", line 828, in do_handshake self._sslobj.do_handshake() SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:727) In many cases this error causes a switch of the active MGR node. But there's another impact on the Ceph Dashboard directly that hangs completely when this error is logged. Any advise how to fix this issue is appreciated. THX

1 0

nfs ganesha rgw write errors
by Marc Roos 18 Nov '19

18 Nov '19

Hi Daniel, I am able to mount the buckets with your config, however when I try to write something, my logs get a lot of these errors: svc_732] nfs4_Errno_verbose :NFS4 :CRIT :Error I/O error in nfs4_write_cb converted to NFS4ERR_IO but was set non-retryable Any chance you know how to resolve this?

2 1