- ceph-users - lists.ceph.io

osdmaps not trimmed until ceph-mon's restarted (if cluster has a down osd)
by Dan van der Ster 09 Dec '19

09 Dec '19

Hi Joao, I might have found the reason why several of our clusters (and maybe Bryan's too) are getting stuck not trimming osdmaps. It seems that when an osd fails, the min_last_epoch_clean gets stuck forever (even long after HEALTH_OK), until the ceph-mons are restarted. I've updated the ticket: https://tracker.ceph.com/issues/41154 Cheers, Dan

4 4

Re: RGW listing millions of objects takes too much time
by Robert LeBlanc 09 Dec '19

09 Dec '19

On Mon, Dec 9, 2019 at 7:47 AM Arash Shams <ara4sh(a)hotmail.com> wrote: > Dear All, > > I have almost 30 million objects and I want to list them and index them > somewhere else, > Im using boto3 with continuation Marker but it takes almost 9 hours > > can I run it in multiple threads to make it faster? what solution do you > suggest to speedup this process, > > > Thanks > I've thought about indexing objects elsewhere as well. One thought I had was hooking into the HTTP flow where a PUT or DEL would update the objects in some kind of database (async of course). We could also gather stats with GET and POST. Initially, my thoughts were to hook into haproxy since we already use it, but possibly RGW if that is an option. That way it would always be up to date and not have to do big scans on the buckets (our buckets would not perform well with this). I haven't actually gotten to the implementation phase of this idea. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

1 0

RGW listing millions of objects takes too much time
by Arash Shams 09 Dec '19

09 Dec '19

Dear All, I have almost 30 million objects and I want to list them and index them somewhere else, Im using boto3 with continuation Marker but it takes almost 9 hours can I run it in multiple threads to make it faster? what solution do you suggest to speedup this process, Thanks

1 0

OSD state<Start>: transitioning to Stray
by Thomas Schneider 09 Dec '19

09 Dec '19

Hi, I had a failure on 2 of 7 OSD nodes. This caused a server reboot and unfortunately the cluster network failed to come up. This resulted in many OSD down situation. I decided to stop all services (OSD, MGR, MON) and to start them sequentially. Now I have multiple OSD marked as down although the service is running. None of these down OSDS is connected to the 2 nodes with failure. In the OSD logs I can see multiple entries like this: 2019-12-09 11:13:10.378 7f9a372fb700 1 osd.374 pg_epoch: 493189 pg[11.1992( v 457986'92619 (303558'88266,457986'92619] local-lis/les=466724/466725 n=4107 ec=8346/8346 lis/c 466724/466724 les/c/f 466725/466725/176266 468956/493184/468423) [203,412] r=-1 lpr=493184 pi=[466724,493184)/1 crt=457986'92619 lcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray I tried to restart the impacted OSD w/o success, means the relevant OSD is still marked as down. Is there a procedure to overcome this issue, means getting all OSD up? THX

2 4

Multi-site RadosGW with multiple placement targets
by Tobias Urdin 09 Dec '19

09 Dec '19

Hello, I'm trying to wrap my head around how having a multi-site (two zones in one zonegroup) with multiple placement targets but only wanting to replicate some placement targets would work. Can you setup a zonegroup with two zones and it would only replicate the placement targets that the other side has? For example if we have three placement targets in the zonegroup, one zone got all three placement targets and the other zone only has two of them, would it only replicate the two? Best regards

3 3

ceph public network definition
by Frank R 07 Dec '19

07 Dec '19

Is the public_network definition in ceph.conf just used to determine which interface or IP to use for the public network or does it need to need to encompass the public ip addresses of all cluster nodes? Specifically, can the public_network be defined differently for different OSD nodes as long as they have layer 3 connectivity? thx Frank

1 0

rbd_open_by_id crash when connection timeout
by yangjun＠cmss.chinamobile.com 07 Dec '19

07 Dec '19

Hi Jason, dongsheng I found a problem using rbd_open_by_id when connection timeout(errno = 110, ceph version 12.2.8, there is no change about rbd_open_by_id in master branch). int r = ictx->state->open(false); if (r < 0) { // r = -110 delete ictx; // crash，the stack is shown below： } else { *image = (rbd_image_t)ictx; } Stack is ： (gdb) bt Program terminated with signal 11, Segmentation fault. #0 Mutex::Lock (this=this@entry=0x8, no_lockdep=no_lockdep@entry=false) at /var/lib/jenkins/workspace/ceph_L/ceph/src/common/Mutex.cc:92 #1 0x0000ffff765eb814 in Locker (m=..., this=<synthetic pointer>) at /var/lib/jenkins/workspace/ceph_L/ceph/src/common/Mutex.h:115 #2 PerfCountersCollection::remove (this=0x0, l=0xffff85bf0588 <main_arena+88>) at /var/lib/jenkins/workspace/ceph_L/ceph/src/common/perf_counters.cc:62 #3 0x0000ffff757f26c4 in librbd::ImageCtx::perf_stop (this=0x2065d2f0) at /var/lib/jenkins/workspace/ceph_L/ceph/src/librbd/ImageCtx.cc:426 #4 0x0000ffff757f6e48 in librbd::ImageCtx::~ImageCtx (this=0x2065d2f0, __in_chrg=<optimized out>) at /var/lib/jenkins/workspace/ceph_L/ceph/src/librbd/ImageCtx.cc:239 #5 0x0000ffff757e8e70 in rbd_open_by_id (p=p@entry=0x2065e780, id=id@entry=0xffff85928bb4 "20376aa706c0", image=image@entry=0xffff75b9b0b8, snap_name=snap_name@entry=0x0) at /var/lib/jenkins/workspace/ceph_L/ceph/src/librbd/librbd.cc:2692 #6 0x0000ffff75aed524 in __pyx_pf_3rbd_5Image___init__ ( __pyx_v_read_only=0xffff85f02bb8 <_Py_ZeroStruct>, __pyx_v_snapshot=0xffff85f15938 <_Py_NoneStruct>, __pyx_v_by_id=0xffff85f02ba0 <_Py_TrueStruct>, __pyx_v_name=0xffff85928b90, __pyx_v_ioctx=0xffff7f0283d0, __pyx_v_self=0xffff75b9b0a8) at /var/lib/jenkins/workspace/ceph_L/ceph/build/src/pybind/rbd/pyrex/rbd.c:12662 #7 __pyx_pw_3rbd_5Image_1__init__ (__pyx_v_self=0xffff75b9b0a8, __pyx_args=<optimized out>, __pyx_kwds=<optimized out>) at /var/lib/jenkins/workspace/ceph_L/ceph/build/src/pybind/rbd/pyrex/rbd.c:12378 #8 0x0000ffff85e02924 in type_call () from /lib64/libpython2.7.so.1.0 #9 0x0000ffff85dac254 in PyObject_Call () from /lib64/libpython2.7.so.1.0 Meanwhile, I have sone questions about rbd_open, if state->open failed（return value r < 0）, Is there a risk of memory leaks about ictx? extern "C" int rbd_open(rados_ioctx_t p, const char *name, rbd_image_t *image, const char *snap_name) { librados::IoCtx io_ctx; librados::IoCtx::from_rados_ioctx_t(p, io_ctx); TracepointProvider::initialize<tracepoint_traits>(get_cct(io_ctx)); librbd::ImageCtx *ictx = new librbd::ImageCtx(name, "", snap_name, io_ctx, false); tracepoint(librbd, open_image_enter, ictx, ictx->name.c_str(), ictx->id.c_str(), ictx->snap_name.c_str(), ictx->read_only); int r = ictx->state->open(false); if (r >= 0) { // if r < 0, Is there a risk of memory leaks? *image = (rbd_image_t)ictx; } tracepoint(librbd, open_image_exit, r); return r; } Can you give me some advice? many thanks. yangjun(a)cmss.chinamobile.com

3 6

Upgrade from Jewel to Nautilus
by 徐蕴 07 Dec '19

07 Dec '19

Hello, We are planning to upgrade our cluster from Jewel to Nautilus. From my understanding, leveldb of monitor and filestore of OSDs will not be converted to rocketdb and bluestore automatically. So do you suggest to convert them manually after upgrading software? Is there any document or guidance available? Br, Xu Yun

2 2

Starting service rbd-target-api fails
by Thomas Schneider 06 Dec '19

06 Dec '19

Hi, I want to setup Ceph iSCSI Gateway and I follow this <https://docs.ceph.com/docs/master/rbd/iscsi-overview/> documentation. In step "Setup" of process "Configuring the iSCSI target using the command line interface <https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/>" I cannot start service rbd-target-api. There's no error message in status or anywhere else: root@ld5505:~# systemctl status rbd-target-api _ rbd-target-api.service - Ceph iscsi target configuration API Loaded: loaded (/lib/systemd/system/rbd-target-api.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Wed 2019-12-04 13:47:51 CET; 3min 16s ago Process: 4143457 ExecStart=/usr/bin/rbd-target-api (code=exited, status=1/FAILURE) Main PID: 4143457 (code=exited, status=1/FAILURE) Dec 04 13:47:51 ld5505 systemd[1]: rbd-target-api.service: Service RestartSec=100ms expired, scheduling restart. Dec 04 13:47:51 ld5505 systemd[1]: rbd-target-api.service: Scheduled restart job, restart counter is at 3. Dec 04 13:47:51 ld5505 systemd[1]: Stopped Ceph iscsi target configuration API. Dec 04 13:47:51 ld5505 systemd[1]: rbd-target-api.service: Start request repeated too quickly. Dec 04 13:47:51 ld5505 systemd[1]: rbd-target-api.service: Failed with result 'exit-code'. Dec 04 13:47:51 ld5505 systemd[1]: Failed to start Ceph iscsi target configuration API. Can you please advise how to troubleshoot this issue? THX

2 4

What are the performance implications 'ceph fs set cephfs allow_new_snaps true'?
by Marc Roos 06 Dec '19

06 Dec '19

I recently enabled this and now my rsyncs are taking hours and hours longer.

1 0