Yes. After the time-out of 600 secs the OSDs got marked down, all PGs got remapped and recovery/rebalancing started as usual. In the past, I did service on servers with the flag noout set and would expect that mon_osd_down_out_subtree_limit=host has the same effect when shutting down an entire host. Unfortunately, in my case these two settings behave differently.
If I understand the documentation correctly, the OSDs should not get marked out automatically.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Anthony D'Atri <anthony.datri(a)gmail.com>
Sent: 14 July 2020 04:32:05
To: Frank Schilder
Subject: Re: [ceph-users] mon_osd_down_out_subtree_limit not working?
Did it start rebalancing?
> On Jul 13, 2020, at 4:29 AM, Frank Schilder <frans(a)dtu.dk> wrote:
>
> if I shut down all OSDs on this host, these OSDs should not be marked out automatically after mon_osd_down_out_interval(=600) seconds. I did a test today and, unfortunately, the OSDs do get marked as out. Ceph status was showing 1 host down as expected.
> If I may ask, which version of the virtio drivers do you use?
https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-vir…
Looks like virtio-win-0.1.185.*
> And do you use caching on libvirt driver level?
In the ONE interface, we use
DISK = [ driver = "raw" , cache = "none"]
which translates to
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
in the XML. We have no qemu settings in the ceph.conf. Looks like caching is disabled. Not sure if this is the recommended way though and why caching is disabled by default.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: André Gemünd <andre.gemuend(a)scai.fraunhofer.de>
Sent: 13 July 2020 11:18
To: Frank Schilder
Subject: Re: [ceph-users] Re: Poor Windows performance on ceph RBD.
If I may ask, which version of the virtio drivers do you use?
And do you use caching on libvirt driver level?
Greetings
André
----- Am 13. Jul 2020 um 10:43 schrieb Frank Schilder frans(a)dtu.dk:
>> > To anyone who is following this thread, we found a possible explanation for
>> > (some of) our observations.
>
>> If someone is following this, they probably want the possible
>> explanation and not the knowledge of you having the possible
>> explanation.
>
>> So you are saying if you do eg. a core installation (without gui) of
>> 2016/2019 disable all services. The fio test results are signficantly
>> different to eg. a centos 7 vm doing the same fio test? Are you sure
>> this is not related to other processes writing to disk?
>
> Right, its not an explanation but rather a further observation. We don't really
> have an explanation yet.
>
> Its an identical installation of both server versions, same services configured.
> Our operators are not really into debugging Windows, that's why we were asking
> here. Their hypothesis is, that the VD driver for accessing RBD images has
> problems with Windows servers newer than 2016. I'm not a Windows guy, so can't
> really comment on this.
>
> The test we do is a simple copy-test of a single 10g file and we monitor the
> transfer speed. This info was cut out of this e-mail, the original report for
> reference is:
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/ANHJQZLJT4…
> .
>
> We are very sure that it is not related to other processes writing to disk, we
> monitor that too. There is also no competition on the RBD pool at the time of
> testing.
>
> Best regards,
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> ________________________________________
> From: Marc Roos <M.Roos(a)f1-outsourcing.eu>
> Sent: 13 July 2020 10:24
> To: ceph-users; Frank Schilder
> Subject: RE: [ceph-users] Re: Poor Windows performance on ceph RBD.
>
>>> To anyone who is following this thread, we found a possible
> explanation for
>>> (some of) our observations.
>
> If someone is following this, they probably want the possible
> explanation and not the knowledge of you having the possible
> explanation.
>
> So you are saying if you do eg. a core installation (without gui) of
> 2016/2019 disable all services. The fio test results are signficantly
> different to eg. a centos 7 vm doing the same fio test? Are you sure
> this is not related to other processes writing to disk?
>
>
>
> -----Original Message-----
> From: Frank Schilder [mailto:frans@dtu.dk]
> Sent: maandag 13 juli 2020 9:28
> To: ceph-users(a)ceph.io
> Subject: [ceph-users] Re: Poor Windows performance on ceph RBD.
>
> To anyone who is following this thread, we found a possible explanation
> for (some of) our observations.
>
> We are running Windows servers version 2016 and 2019 as storage servers
> exporting data on an rbd image/disk. We recently found that Windows
> server 2016 runs fine. It is still not as fast as Linux + SAMBA share on
> an rbd image (ca. 50%), but runs with a reasonable sustained bandwidth.
> With Windows server 2019, however, we observe near-complete stall of
> file transfers and time-outs using standard copy tools (robocopy). We
> don't have an explanation yet and are downgrading Windows servers where
> possible.
>
> If anyone has a hint what we can do, please let us know.
>
> Best regards,
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
> email to ceph-users-leave(a)ceph.io
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Dipl.-Inf. André Gemünd, Leiter IT / Head of IT
Fraunhofer-Institute for Algorithms and Scientific Computing
andre.gemuend(a)scai.fraunhofer.de
Tel: +49 2241 14-2193
/C=DE/O=Fraunhofer/OU=SCAI/OU=People/CN=Andre Gemuend
Hi all,
I want to use cephfs-shell dealing with operations like directory creation,
instead of mounting the root directory and create manually. But I get
errors
when I execute the command 'cephfs-shell'.
Traceback (most recent call last):
File "./cephfs-shell", line 9, in <module>
import cephfs as libcephfs
ModuleNotFoundError: No module named 'cephfs'
CentOS7 use python2 and I've already installed python3 in the system.
What else should I do, reinstall libcephfs?
Thanks
Hi,
We use the official Ceph RPM repository (http://download.ceph.com/rpm-
nautilus/el7) for installing packages on the client nodes running
CentOS7.
But we noticed today that the repo only provides the latest version
(2:14.2.10-0.el7) of nautilus so that we couldn't install an older
(2:14.2.7-0.el7) version of the ceph-common package.
The ceph.repo file contains:
---
[Ceph]
name=Ceph packages for $basearch
baseurl=http://download.ceph.com/rpm-nautilus/el7/$basearch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
---
and
---
$ yum clean all
$ yum makecache
$ yum list --disablerepo=* --enablerepo=Ceph --showduplicates ceph-
common
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
Available Packages
ceph-common.x86_64 2:14.2.10-
0.el7 Ceph
---
Does it mean that the official repo no longer provides RPM packages for
older versions? Thanks!
Cheers, Hong
--
Hurng-Chun (Hong) Lee, PhD
ICT manager
Donders Institute for Brain, Cognition and Behaviour,
Centre for Cognitive Neuroimaging
Radboud University Nijmegen
e-mail: h.lee(a)donders.ru.nl
tel: +31(0) 243610977
web: http://www.ru.nl/donders/
Hi,
I’ve been getting errors in the NFS section of the web interface. I’ve just tried upgrading to 15.2.4 to see if that helped but no joy.
The initial NFS page loads OK and when I click Add, a form loads. However, when this form attempts to update its values, I get a red box informing me that the server returned an error 500. it makes four HTTP calls - to daemon, clients, filesystems and fsals - these all fail.
Any suggestions on what might be wrong or where some useful logs might be?
Ta,
Will
Hi All,
I'm investigating what appears to be a bug in RGW stats. This is a brand
new cluster running 15.2.3
One of our customers reached out, saying they were hitting their quota (S3
error: 403 (QuotaExceeded)). The user-wide max_objects quota we set is 50
million objects, so this would be impossible since the entire cluster isn't
even close to 50 million objects yet:
[root@os1 ~]# ceph status | grep objects
objects: 7.58M objects, 6.8 TiB
The customer in question has three buckets, and if I query the bucket
stats, the total number of objects for all 3 buckets comes to about ~372k:
[root@os1 ~]# radosgw-admin bucket stats --bucket=df-fs1 | grep num_objects
"num_objects": 324880
[root@os1 ~]# radosgw-admin bucket stats --bucket=df-oldrepo | grep
num_objects
"num_objects": 47476
[root@os1 ~]# radosgw-admin bucket stats --bucket=df-test | grep num_objects
"num_objects": 1
But things get interesting when I query the user stats:
Hello,
I'm trying to adopt an existing cluster.
The cluster consists of 5 converged (mon, mgr, osd, mds on same host)
servers running Octopus 15.2.4.
I've followed the guide:
https://docs.ceph.com/docs/octopus/cephadm/adoption/
Adopting the first mon I've got the following problem:
root@mulberry:/home/toga# cephadm adopt --style legacy --name mon.mulberry
INFO:cephadm:Pulling latest docker.io/ceph/ceph:v15 container...
INFO:cephadm:Stopping old systemd unit ceph-mon@mulberry...
INFO:cephadm:Disabling old systemd unit ceph-mon@mulberry...
INFO:cephadm:Moving data...
INFO:cephadm:Chowning content...
Traceback (most recent call last):
File "/usr/sbin/cephadm", line 4761, in <module>
r = args.func()
File "/usr/sbin/cephadm", line 1162, in _default_image
return func()
File "/usr/sbin/cephadm", line 3241, in command_adopt
command_adopt_ceph(daemon_type, daemon_id, fsid);
File "/usr/sbin/cephadm", line 3387, in command_adopt_ceph
call_throws(['chown', '-c', '-R', '%d.%d' % (uid, gid), data_dir_dst])
File "/usr/sbin/cephadm", line 844, in call_throws
out, err, ret = call(command, **kwargs)
File "/usr/sbin/cephadm", line 784, in call
message = message_b.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position
1023: unexpected end of data
In `cephadm ls` the old mon is gone and the new is present:
{
"style": "cephadm:v1",
"name": "mon.mulberry",
"fsid": "74307e84-e1fe-4706-8312-fe47703928a1",
"systemd_unit":
"ceph-74307e84-e1fe-4706-8312-fe47703928a1(a)mon.mulberry",
"enabled": false,
"state": "stopped",
"container_id": null,
"container_image_name": null,
"container_image_id": null,
"version": null,
"started": null,
"created": null,
"deployed": null,
"configured": null
}
But there is no container running.
How can I resolve this?
Regards,
Tobias
Thanks for this info -- adding it to our list of reasons never to use
FileStore again.
In your case, are you able to migrate?
On Tue, Jul 14, 2020 at 3:13 PM Eric Smith <Eric.Smith(a)vecima.com> wrote:
>
> FWIW Bluestore is not affected by this problem!
>
> -----Original Message-----
> From: Eric Smith <Eric.Smith(a)vecima.com>
> Sent: Saturday, July 11, 2020 6:40 AM
> To: ceph-users(a)ceph.io
> Subject: [ceph-users] Re: Luminous 12.2.12 - filestore OSDs take an hour to boot
>
> It does appear that long file names and filestore seem to be a real problem. We have a cluster where 99% of the objects have names longer than N (220+?) characters such that it truncates the file name (as seen below with "_<sha-sum>_0_long") and stores the full object name in xattrs for the object. During boot the OSD goes out to lunch for increasing amounts of time based on the number of objects on disk you have that meet this criteria (With 2.4 million ish objects that meet this criteria, the OSD takes over an hour to boot). I plan on testing this same scenario with BlueStore to see if it's also susceptible to these boot / read issues.
>
> Eric
>
> -----Original Message-----
> From: Eric Smith <Eric.Smith(a)vecima.com>
> Sent: Friday, July 10, 2020 1:46 PM
> To: ceph-users(a)ceph.io
> Subject: [ceph-users] Re: Luminous 12.2.12 - filestore OSDs take an hour to boot
>
> For what it's worth - all of our objects are generating LONG named object files like so...
>
> \uABCD\ucontent.\srecording\swzdchd\u\utnda-trg-1008007-wzdchd-216203706303281120-230932949-1593482400-159348660000000001\swzdchd\u\utpc2-tp1-1008007-wzdchd-216203706303281120-230932949-1593482400-159348660000000001\u\uwzdchd3._0bfd7c716b839cb7b3ad_0_long
>
> Does this matter? AFAICT it sees this as a long file name and has to lookup the object name in the xattrs ? Is that bad?
>
> -----Original Message-----
> From: Eric Smith <Eric.Smith(a)vecima.com>
> Sent: Friday, July 10, 2020 6:59 AM
> To: ceph-users(a)ceph.io
> Subject: [ceph-users] Luminous 12.2.12 - filestore OSDs take an hour to boot
>
> I have a cluster running Luminous 12.2.12 with Filestore and it takes my OSDs somewhere around an hour to start (They do start successfully - eventually). I have the following log entries that seem to show the OSD process attempting to descend into the PG directory on disk and create an object list of some sort:
>
> 2020-07-09 18:29:28.017207 7f3b680afd80 20 osd.1 137390 clearing temps in 8.14ads3_head pgid 8.14ads3
> 2020-07-09 18:29:28.017211 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is 8 shard is 3 pgid 8.14ads3
> 2020-07-09 18:29:28.017213 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5020): first checking temp pool
> 2020-07-09 18:29:28.017215 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is -10 shard is 3 pgid 8.14ads3
> 2020-07-09 18:29:28.017221 7f3b680afd80 20 _collection_list_partial start:GHMIN end:GHMAX-64 ls.size 0
> 2020-07-09 18:29:28.017263 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) objects: []
> 2020-07-09 18:29:28.017268 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5028): fall through to non-temp collection, start 3#-1:00000000::::0#
> 2020-07-09 18:29:28.017272 7f3b680afd80 20 _collection_list_partial start:3#-1:00000000::::0# end:GHMAX-64 ls.size 0
> 2020-07-09 18:29:28.038124 7f3b680afd80 20 list_by_hash_bitwise prefix D
> 2020-07-09 18:29:28.058679 7f3b680afd80 20 list_by_hash_bitwise prefix DA
> 2020-07-09 18:29:28.069432 7f3b680afd80 20 list_by_hash_bitwise prefix DA4
> 2020-07-09 18:29:29.789598 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000074
> 2020-07-09 18:29:29.789634 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0
> 2020-07-09 18:29:29.789639 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed
> 2020-07-09 18:29:29.789641 7f3b51a87700 10 journal commit_start nothing to do
> 2020-07-09 18:29:29.789663 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000
> 2020-07-09 18:29:34.789815 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000109
> 2020-07-09 18:29:34.789898 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0
> 2020-07-09 18:29:34.789902 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed
> 2020-07-09 18:29:34.789906 7f3b51a87700 10 journal commit_start nothing to do
> 2020-07-09 18:29:34.789939 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000
> 2020-07-09 18:29:38.651689 7f3b680afd80 20 list_by_hash_bitwise prefix DA41
> 2020-07-09 18:29:39.790069 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000128
> 2020-07-09 18:29:39.790090 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0
> 2020-07-09 18:29:39.790092 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed
> 2020-07-09 18:29:39.790093 7f3b51a87700 10 journal commit_start nothing to do
> 2020-07-09 18:29:39.790102 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000
> 2020-07-09 18:29:44.790200 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000095
> 2020-07-09 18:29:44.790256 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0
> 2020-07-09 18:29:44.790265 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed
> 2020-07-09 18:29:44.790268 7f3b51a87700 10 journal commit_start nothing to do
> 2020-07-09 18:29:44.790286 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000
> 2020-07-09 18:29:49.790353 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000066
> 2020-07-09 18:29:49.790374 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0
> 2020-07-09 18:29:49.790376 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed
> 2020-07-09 18:29:49.790378 7f3b51a87700 10 journal commit_start nothing to do
> 2020-07-09 18:29:49.790387 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000
> 2020-07-09 18:29:50.564479 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000
> 2020-07-09 18:29:50.564501 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000 ob 3#8:b5280000::::head#
> 2020-07-09 18:29:50.564508 7f3b680afd80 20 list_by_hash_bitwise prefix DA41002A
>
> Any idea what's going on here? I can run a find of every file on the filesystem in under 12 minutes so I'm not sure what's taking so long.
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
FWIW Bluestore is not affected by this problem!
-----Original Message-----
From: Eric Smith <Eric.Smith(a)vecima.com>
Sent: Saturday, July 11, 2020 6:40 AM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Luminous 12.2.12 - filestore OSDs take an hour to boot
It does appear that long file names and filestore seem to be a real problem. We have a cluster where 99% of the objects have names longer than N (220+?) characters such that it truncates the file name (as seen below with "_<sha-sum>_0_long") and stores the full object name in xattrs for the object. During boot the OSD goes out to lunch for increasing amounts of time based on the number of objects on disk you have that meet this criteria (With 2.4 million ish objects that meet this criteria, the OSD takes over an hour to boot). I plan on testing this same scenario with BlueStore to see if it's also susceptible to these boot / read issues.
Eric
-----Original Message-----
From: Eric Smith <Eric.Smith(a)vecima.com>
Sent: Friday, July 10, 2020 1:46 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Luminous 12.2.12 - filestore OSDs take an hour to boot
For what it's worth - all of our objects are generating LONG named object files like so...
\uABCD\ucontent.\srecording\swzdchd\u\utnda-trg-1008007-wzdchd-216203706303281120-230932949-1593482400-159348660000000001\swzdchd\u\utpc2-tp1-1008007-wzdchd-216203706303281120-230932949-1593482400-159348660000000001\u\uwzdchd3._0bfd7c716b839cb7b3ad_0_long
Does this matter? AFAICT it sees this as a long file name and has to lookup the object name in the xattrs ? Is that bad?
-----Original Message-----
From: Eric Smith <Eric.Smith(a)vecima.com>
Sent: Friday, July 10, 2020 6:59 AM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Luminous 12.2.12 - filestore OSDs take an hour to boot
I have a cluster running Luminous 12.2.12 with Filestore and it takes my OSDs somewhere around an hour to start (They do start successfully - eventually). I have the following log entries that seem to show the OSD process attempting to descend into the PG directory on disk and create an object list of some sort:
2020-07-09 18:29:28.017207 7f3b680afd80 20 osd.1 137390 clearing temps in 8.14ads3_head pgid 8.14ads3
2020-07-09 18:29:28.017211 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is 8 shard is 3 pgid 8.14ads3
2020-07-09 18:29:28.017213 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5020): first checking temp pool
2020-07-09 18:29:28.017215 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is -10 shard is 3 pgid 8.14ads3
2020-07-09 18:29:28.017221 7f3b680afd80 20 _collection_list_partial start:GHMIN end:GHMAX-64 ls.size 0
2020-07-09 18:29:28.017263 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) objects: []
2020-07-09 18:29:28.017268 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5028): fall through to non-temp collection, start 3#-1:00000000::::0#
2020-07-09 18:29:28.017272 7f3b680afd80 20 _collection_list_partial start:3#-1:00000000::::0# end:GHMAX-64 ls.size 0
2020-07-09 18:29:28.038124 7f3b680afd80 20 list_by_hash_bitwise prefix D
2020-07-09 18:29:28.058679 7f3b680afd80 20 list_by_hash_bitwise prefix DA
2020-07-09 18:29:28.069432 7f3b680afd80 20 list_by_hash_bitwise prefix DA4
2020-07-09 18:29:29.789598 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000074
2020-07-09 18:29:29.789634 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0
2020-07-09 18:29:29.789639 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed
2020-07-09 18:29:29.789641 7f3b51a87700 10 journal commit_start nothing to do
2020-07-09 18:29:29.789663 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000
2020-07-09 18:29:34.789815 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000109
2020-07-09 18:29:34.789898 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0
2020-07-09 18:29:34.789902 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed
2020-07-09 18:29:34.789906 7f3b51a87700 10 journal commit_start nothing to do
2020-07-09 18:29:34.789939 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000
2020-07-09 18:29:38.651689 7f3b680afd80 20 list_by_hash_bitwise prefix DA41
2020-07-09 18:29:39.790069 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000128
2020-07-09 18:29:39.790090 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0
2020-07-09 18:29:39.790092 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed
2020-07-09 18:29:39.790093 7f3b51a87700 10 journal commit_start nothing to do
2020-07-09 18:29:39.790102 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000
2020-07-09 18:29:44.790200 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000095
2020-07-09 18:29:44.790256 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0
2020-07-09 18:29:44.790265 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed
2020-07-09 18:29:44.790268 7f3b51a87700 10 journal commit_start nothing to do
2020-07-09 18:29:44.790286 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000
2020-07-09 18:29:49.790353 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000066
2020-07-09 18:29:49.790374 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0
2020-07-09 18:29:49.790376 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed
2020-07-09 18:29:49.790378 7f3b51a87700 10 journal commit_start nothing to do
2020-07-09 18:29:49.790387 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000
2020-07-09 18:29:50.564479 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000
2020-07-09 18:29:50.564501 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000 ob 3#8:b5280000::::head#
2020-07-09 18:29:50.564508 7f3b680afd80 20 list_by_hash_bitwise prefix DA41002A
Any idea what's going on here? I can run a find of every file on the filesystem in under 12 minutes so I'm not sure what's taking so long.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io