- ceph-users - lists.ceph.io

Recoveries without any misplaced objects?
by Hector Martin 29 May '23

29 May '23

Hi, I'm watching a cluster finish a bunch of backfilling, and I noticed that quite often PGs end up with zero misplaced objects, even though they are still backfilling. Right now the cluster is down to 6 backfilling PGs: data: volumes: 1/1 healthy pools: 6 pools, 268 pgs objects: 18.79M objects, 29 TiB usage: 49 TiB used, 25 TiB / 75 TiB avail pgs: 262 active+clean 6 active+remapped+backfilling But there are no misplaced objects, and the misplaced column in `ceph pg dump` is zero for all PGs. If I do a `ceph pg dump_json`, I can see `num_objects_recovered` increasing for these PGs... but the misplaced count is still 0. Is there something else that would cause recoveries/backfills other than misplaced objects? Or perhaps there is a bug somewhere causing the misplaced object count to be misreported as 0 sometimes? # ceph -v ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) - Hector

1 0

[Ceph | Quency ]The scheduled snapshots are not getting created till we create a manual backup.
by Lokendra Rathour 29 May '23

29 May '23

Hi Team, *Problem:* Create scheduled snapshots of the ceph subvolume. *Expected Result:* The scheduled snapshots should be created at the given scheduled time. *Actual Result:* The scheduled snapshots are not getting created till we create a manual backup. *Description:* *Ceph version: 17(quincy)* OS: Centos/Almalinux The scheduled snapshot creation is not working and we were only able to see the following logs in the file "ceph-mgr.storagenode3.log": *2023-05-29T04:59:35.101+0000 7f4cd3ad8700 0 [snap_schedule INFO mgr_util] scanning for idle connections..* *2023-05-29T04:59:35.101+0000 7f4cd3ad8700 0 [snap_schedule DEBUG mgr_util] fs_name (cephfs) connections ([])* *2023-05-29T04:59:35.101+0000 7f4cd3ad8700 0 [snap_schedule INFO mgr_util] cleaning up connections: [* The command which we were executing to add the snapshot schedule: *ceph fs snap-schedule add /volumes/<subvolume_group>/<subvolume> <frequency_of_snapshot> <Timeinformat_yyyymmddTHH:MM:ss>* *eg.* *ceph fs snap-schedule add /volumes/xyz/test_restore_53 1h 2023-05-26T11:05:00* We can make sure that the schedule has been created using the following commands: *#ceph fs snap-schedule list / --recursive=true* *#ceph fs snap-schedule status /volumes/xyz/test_restore_53* Even though we created the snapshot schedule, snapshots were not getting created. We then tried creating a manual snapshot for one of the sub-volumes using the following command: *#ceph fs subvolume snapshot create cephfs <subvolume> <snapshot-name> --group_name <subvolume_group>* *eg. ceph fs subvolume snapshot create cephfs test_restore_53 snapshot-1 --group_name xyz* To check the snapshots created we can use the following command: *ceph fs subvolume snapshot ls cephfs <subvolume> <snapshot-name> <subvolume_group>* *eg. ceph fs subvolume snapshot ls cephfs test_restore_53 snapshot-1 xyz* To delete the manually created snapshot: *ceph fs subvolume snapshot rm cephfs <subvolume> <snapshot-name> <subvolume_group>* *eg. ceph fs subvolume snapshot rm cephfs test_restore_53 snapshot-1 xyz* To our surprise, the scheduled snapshots started working. We also applied the retention policy and seems to be working fine. We re-tested this understanding for another subvolume. And the scheduled snapshots only started once we triggered a manual snapshot. Could you please help us out with this? -- ~ Lokendra skype: lokendrarathour

2 1

Creating a bucket with bucket constructor in Ceph v16.2.7
by Ramin Najjarbashi 29 May '23

29 May '23

Hi I'm currently using Ceph version 16.2.7 and facing an issue with bucket creation in a multi-zone configuration. My setup includes two zone groups: ZG1 (Master) and ZG2, with one zone in each zone group (zone-1 in ZG1 and zone-2 in ZG2). The objective is to create buckets in a specific zone group (ZG2) using the bucket constructor. However, despite setting the desired zone group (abrak) in the request, the bucket is still being created in the master zone group (ZG1). I have defined the following endpoint pattern for each zone group: s3.{zg}.mydomain.com I am using the s3cmd client to interact with the Ceph cluster. I have ensured that I provide the necessary endpoint and region information while executing the bucket creation command. Despite my efforts, the bucket consistently gets created in ZG1 instead of ZG2. - Ceph Version: 16.2.7 - Zone Group 1 (ZG1) Endpoint: http://s3.zonegroup1.mydomain.com - Zone Group 2 (ZG2) Endpoint: http://s3.zonegroup2.mydomain.com - Desired Bucket Creation Region: zg2-api-name have reviewed the Ceph documentation and made necessary configuration changes, but I have not been able to achieve the desired result.I kindly request your assistance in understanding why the bucket constructor is not honoring the specified region and always defaults to ZG1. I would greatly appreciate any insights, recommendations, or potential solutions to resolve this issue. Thank you for your time and support. --------------------------------- Here are the details of my setup: --------------------------------- ```sh s3cmd --region zg2-api-name mb s3://test-zg2s3cmd info s3://test-zg2s3://test-zg2/ (bucket): Location: zg2-api-name Payer: BucketOwner Expiration Rule: none Policy: none CORS: none ACL: development: FULL_CONTROL ``` this is my config file: ```ini [default] access_key = <KEY> secret_key = <SECRET> host_base = s3.zonegroup1.mydomain.com host_bucket = s3.%(location)s.mydomain.com #host_bucket = %(bucket)s.s3.zonegroup1.mydomain.com #host_bucket = s3.%(location)s.mydomain.com #host_bucket = s3.%(region)s.mydomain.com bucket_location = zg1-api-name use_https = False ``` Zonegroup configuration for the `zonegroup1` region: ```json { "id": "fb3f818a-ca9b-4b12-b431-7cdcd80006d", "name": "zg1-api-name", "api_name": "zg1-api-name", "is_master": "false", "endpoints": [ "http://s3.zonegroup1.mydomain.com", ], "hostnames": [ "s3.zonegroup1.mydomain.com", ], "hostnames_s3website": [ "s3-website.zonegroup1.mydomain.com", ], "master_zone": "at2-stg-zone", "zones": [ { "id": "at2-stg-zone", "name": "at2-stg-zone", "endpoints": [ "http://s3.zonegroup1.mydomain.com" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "fa2f8194-4a9d-4b98-b411-9cdcd1e5506a", "sync_policy": { "groups": [] } } ``` Zonegroup configuration for the `zonegroup2` region: ```json { "id": "a513d60c-44a2-4289-a23d-b7a511be6ee4", "name": "zg2-api-name", "api_name": "zg2-api-name", "is_master": "false", "endpoints": [ "http://s3.zonegroup2.mydomain.com" ], "hostnames": [ "s3.zonegroup2.mydomain.com" ], "hostnames_s3website": [], "master_zone": "zonegroup2-sh-1", "zones": [ { "id": "zonegroup2-sh-1", "name": "zonegroup2-sh-1", "endpoints": [ "http://s3.zonegroup2.mydomain.com" ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "fa2f8194-4a9d-4b98-b411-9cdcd1e5506a", "sync_policy": { "groups": [] } } ```

3 4

Can lifecycle transition cause a high load if many buckets do transition at the same time?
by huy nguyen 28 May '23

28 May '23

Hi, I have a plan to use the second storage class for cold data in the near future. But I have a question: if many old buckets with many objects are set to transition at the same time, can Ceph handle it by default, or do I have to configure QoS to avoid the high load? Thanks!

1 0

Ceph iscsi gateway semi deprecation warning?
by Mark Kirkwood 28 May '23

28 May '23

I am looking at using an iscsi gateway in front of a ceph setup. However the warning in the docs is concerning: The iSCSI gateway is in maintenance as of November 2022. This means that it is no longer in active development and will not be updated to add new features. Does this mean I should be wary of using it, or is it simply that it does all the stuff it needs to and no further development is needed? regards Mark

3 2

fail delete "daemon(s) not managed by cephadm"
by farhad kh 28 May '23

28 May '23

hi everyone i have a warning ` 1 stray daemon(s) not managed by cephadm` # ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon mon.apcepfpspsp0111 on host apcepfpspsp0111 not managed by cephadm and in detail : # ceph crash info 2023-01-17T18:25:19.601062Z_4affd3b9-f486-4ae5-801b-ede231e2b624 { "archived": "2023-01-18 19:31:58.200096", "assert_condition": "abort", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/mon/MonitorDBStore.h", "assert_func": "int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)", "assert_line": 355, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7fabeb737700 time 2023-01-17T18:25:19.568046+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/mon/MonitorDBStore.h: 355: ceph_abort_msg(\"failed to write to db\")\n", "assert_thread_name": "ms_dispatch", "backtrace": [ "/lib64/libpthread.so.0(+0x12cf0) [0x7fabf5949cf0]", "gsignal()", "abort()", "(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x197) [0x7fabf799eb5f]", "(MonitorDBStore::apply_transaction(std::shared_ptr<MonitorDBStore::Transaction>)+0x88f) [0x55fe9cc3a68f]", "(Elector::persist_epoch(unsigned int)+0x184) [0x55fe9ccc64c4]", "(ElectionLogic::bump_epoch(unsigned int)+0x5d) [0x55fe9ccccd5d]", "(ElectionLogic::propose_classic_prefix(int, unsigned int)+0x3c7) [0x55fe9ccce8e7]", "(ElectionLogic::propose_classic_handler(int, unsigned int)+0x2b) [0x55fe9cccef9b]", "(Elector::handle_propose(boost::intrusive_ptr<MonOpRequest>)+0x6f5) [0x55fe9ccc6175]", "(Elector::dispatch(boost::intrusive_ptr<MonOpRequest>)+0xcdb) [0x55fe9ccc756b]", "(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x11c2) [0x55fe9cc0aeb2]", "(Monitor::_ms_dispatch(Message*)+0x406) [0x55fe9cc0b5e6]", "(Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5d) [0x55fe9cc3bdad]", "(Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478) [0x7fabf7c18c88]", "(DispatchQueue::entry()+0x50f) [0x7fabf7c160cf]", "(DispatchQueue::DispatchThread::entry()+0x11) [0x7fabf7cdd8f1]", "/lib64/libpthread.so.0(+0x81ca) [0x7fabf593f1ca]", "clone()" ], "ceph_version": "17.2.5", "crash_id": "2023-01-17T18:25:19.601062Z_4affd3b9-f486-4ae5-801b-ede231e2b624", "entity_name": "mon.apcepfpspsp0101", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "ceph-mon", "stack_sig": "74f29b024a3ec0f1395579505c5728e8ad5fd5a77c4a92437d696fa6fd1a8e20", "timestamp": "2023-01-17T18:25:19.601062Z", "utsname_hostname": "apcepfpspsp0101", "utsname_machine": "x86_64", "utsname_release": "5.4.17-2136.310.7.1.el8uek.x86_64", "utsname_sysname": "Linux", "utsname_version": "#2 SMP Wed Aug 17 15:14:08 PDT 2022" } but i have 3 instance from mon modul NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 3/3 2m ago 9M label:mon crash 6/6 9m ago 9M * grafana ?:3000 3/3 2m ago 4M count-per-host:1;label:mon mds.rook 3/3 2m ago 8M apcepfpspsp0101;apcepfpspsp0103;apcepfpspsp0105;count:3 mgr 2/2 2m ago 9M count:2 mon 3/3 2m ago 2h label:mon node-exporter ?:9100 6/6 9m ago 9M * osd 3 2m ago - <unmanaged> osd.cost_capacity 15 9m ago 4M * prometheus ?:9095 3/3 2m ago 9M label:mon whene i want remove this get # ceph orch daemon rm mon.apcepfpspsp0111 --force Error EINVAL: Unable to find daemon(s) ['mon.apcepfpspsp0111'] I don't have such a daemon mon.apcepfpspsp0101 apcepfpspsp0101 running (88m) mon.apcepfpspsp0103 apcepfpspsp0103 running (87m) mon.apcepfpspsp0105 apcepfpspsp0105 running (87m) What should I do to remove this warning?

2 1

cephfs-data-scan with multiple data pools
by Justin Li 26 May '23

26 May '23

Dear All, I'm trying to recover failed MDS metadata by following the link below but having troubles. Thanks in advance. Question1: how to scan 2 data pools with scan_extents (cmd 1). The cmd didn't work with two pools specified. Should I scan one then the other? Question2: As to scan_inodes (cmd 2), should I only specify the first data pool per the document. I'm concerned if the 2nd pool is not scanned then that'll cause metadata loss. *my fs name: cephfs, data pools: cephfs_hdd, cephfs_ssd* cmd 1: cephfs-data-scan scan_extents --filesystem cephfs cephfs_hdd cephfs_ssd cmd 2: cephfs-data-scan scan_inodes --filesystem cephfs cephfs_hdd cephfs-data-scan scan_extents [<data pool> [<extra data pool> ...]]cephfs-data-scan scan_inodes [<data pool>]cephfs-data-scan scan_links Note, the data pool parameters for ‘scan_extents’, ‘scan_inodes’ and ‘cleanup’ commands are optional, and usually the tool will be able to detect the pools automatically. Still you may override this. The ‘scan_extents’ command needs all data pools to be specified,* while ‘scan_inodes’ and ‘cleanup’ commands need only the main data pool.* *https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ <https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/>* -- Best Regards, *Justin Li* IT Support/Systems Administrator *Justin.Li2030(a)Gmail.com <Justin.Li2030(a)Gmail.com>* <http://www.linkedin.com/in/justinli7>

1 0

Unable to online CephFS, MDS segfaults during mds log replay
by Alfred Heisner 26 May '23

26 May '23

Hello, I have a Ceph deployment using CephFS. Recently MDS failed and cannot start. Attempting to start MDS for this filesystem results in nearly immediate segfault in MDS. Logs below. cephfs-journal-tool shows Overall journal integrity state OK root@proxmox-2:/var/log/ceph# cephfs-journal-tool --rank=galaxy:all journal inspect Overall journal integrity: OK Stack dump / log from MDS: -14> 2023-05-26T15:01:09.204-0500 7f27c24b2700 1 mds.0.journaler.mdlog(ro) probing for end of the log -13> 2023-05-26T15:01:09.208-0500 7f27c34b4700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 4194304, expire 4194607, write 4194607, stream_format 1). probing for end of log (from 4194607)... -12> 2023-05-26T15:01:09.208-0500 7f27c34b4700 1 mds.0.journaler.pq(ro) probing for end of the log -11> 2023-05-26T15:01:09.412-0500 7f27c24b2700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 2388235687 (header had 2388213543). recovered. -10> 2023-05-26T15:01:09.412-0500 7f27c34b4700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 4194607 (header had 4194607). recovered. -9> 2023-05-26T15:01:09.412-0500 7f27c34b4700 4 mds.0.purge_queue operator(): open complete -8> 2023-05-26T15:01:09.412-0500 7f27c34b4700 1 mds.0.journaler.pq(ro) set_writeable -7> 2023-05-26T15:01:09.412-0500 7f27c1cb1700 4 mds.0.log Journal 0x200 recovered. -6> 2023-05-26T15:01:09.412-0500 7f27c1cb1700 4 mds.0.log Recovered journal 0x200 in format 1 -5> 2023-05-26T15:01:09.412-0500 7f27c1cb1700 2 mds.0.6403 Booting: 1: loading/discovering base inodes -4> 2023-05-26T15:01:09.412-0500 7f27c1cb1700 0 mds.0.cache creating system inode with ino:0x100 -3> 2023-05-26T15:01:09.412-0500 7f27c1cb1700 0 mds.0.cache creating system inode with ino:0x1 -2> 2023-05-26T15:01:09.416-0500 7f27c24b2700 2 mds.0.6403 Booting: 2: replaying mds log -1> 2023-05-26T15:01:09.416-0500 7f27c24b2700 2 mds.0.6403 Booting: 2: waiting for purge queue recovered 0> 2023-05-26T15:01:09.428-0500 7f27c0caf700 -1 *** Caught signal (Segmentation fault) ** in thread 7f27c0caf700 thread_name:md_log_replay ceph version 17.2.6 (995dec2cdae920da21db2d455e55efbc339bde24) quincy (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f27cd70c140] 2: (EMetaBlob::replay(MDSRank*, LogSegment*, MDPeerUpdate*)+0x66c2) [0x563540fc7372] 3: (EUpdate::replay(MDSRank*)+0x3c) [0x563540fc8abc] 4: (MDLog::_replay_thread()+0x7cb) [0x563540f4d0fb] 5: (MDLog::ReplayThread::entry()+0xd) [0x563540c1fbfd] 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f27cd700ea7] 7: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. What are the safest steps to recovery at this point? Thanks, Al

1 0

Pacific - MDS behind on trimming
by Emmanuel Jaep 26 May '23

26 May '23

Hi, lately, we have had some issues with our MDSs (Ceph version 16.2.10 Pacific). Part of them are related to MDS being behind on trimming. I checked the documentation and found the following information ( https://docs.ceph.com/en/pacific/cephfs/health-messages/) > CephFS maintains a metadata journal that is divided into *log segments*. The length of journal (in number of segments) is controlled by the setting mds_log_max_segments, and when the number of segments exceeds that setting the MDS starts writing back metadata so that it can remove (trim) the oldest segments. If this writeback is happening too slowly, or a software bug is preventing trimming, then this health message may appear. The threshold for this message to appear is controlled by the config option mds_log_warn_factor, the default is 2.0. Some resources on the web (https://www.suse.com/support/kb/doc/?id=000019740) indicated that a solution would be to change the `mds_log_max_segments`. Which I did: ``` ceph --cluster floki tell mds.* injectargs '--mds_log_max_segments=400000' ``` Of course, the warning disappeared, but I have a feeling that I just hid the problem. Pushing a value to 400'000 when the default value is 512 is a lot. Why is the trimming not taking place? How can I troubleshoot this further? Best, Emmanuel

2 1

mds dump inode crashes file system
by Frank Schilder 26 May '23

26 May '23

Hi all, I have an annoying problem with a specific ceph fs client. We have a file server on which we re-export kernel mounts via samba (all mounts with noshare option). On one of these re-exports we have recurring problems. Today I caught it with 2023-05-10T13:39:50.963685+0200 mds.ceph-23 (mds.1) 1761 : cluster [WRN] client.205899841 isn't responding to mclientcaps(revoke), ino 0x20011d3e5cb pending pAsLsXsFscr issued pAsLsXsFscr, sent 61.705410 seconds ago and I wanted to look up what path the inode 0x20011d3e5cb points to. Unfortunately, the command ceph tell "mds.*" dump inode 0x20011d3e5cb crashes an MDS in a way that it restarts itself, but doesn't seem to come back clean (it does not fail over to a stand-by). If I repeat the command above, it crashes the MDS again. Execution on other MDS daemons succeeds, for example: # ceph tell "mds.ceph-24" dump inode 0x20011d3e5cb 2023-05-10T14:14:37.091+0200 7fa47ffff700 0 client.210149523 ms_handle_reset on v2:192.168.32.88:6800/3216233914 2023-05-10T14:14:37.124+0200 7fa4857fa700 0 client.210374440 ms_handle_reset on v2:192.168.32.88:6800/3216233914 dump inode failed, wrong inode number or the inode is not cached The caps recall gets the client evicted at some point but it doesn't manage to come back clean. On a single ceph fs mount point I see this # ls /shares/samba/rit-oil ls: cannot access '/shares/samba/rit-oil': Stale file handle All other mount points are fine, just this one acts up. A "mount -o remount /shares/samba/rit-oil" crashed the entire server and I had to do a cold reboot. On reboot I see this message: https://imgur.com/a/bOSLxBb , which only occurs on this one file server (we are running a few of those). Does this point to a more serious problem, like a file system corruption? Should I try an fs scrub on the corresponding path? Some info about the system: The file server's kernel version is quite recent, updated two weeks ago: $ uname -r 4.18.0-486.el8.x86_64 # cat /etc/redhat-release CentOS Stream release 8 Our ceph cluster is octopus latest and we use the packages from the octopus el8 repo on this server. We have several such shares and they all work fine. It is only on one share where we have persistent problems with the mount point hanging or the server freezing and crashing. After working hours I will try a proper fail of the "broken" MDS to see if I can execute the dump inode command without it crashing everything. In the mean time, any hints would be appreciated. I see that we have an exceptionally large MDS log for the problematic one. Any hint what to look for would be appreciated, it contains a lot from the recovery operations: # pdsh -w ceph-[08-17,23-24] ls -lh "/var/log/ceph/ceph-mds.ceph-??.log" ceph-23: -rw-r--r--. 1 ceph ceph 15M May 10 14:28 /var/log/ceph/ceph-mds.ceph-23.log *** huge *** ceph-24: -rw-r--r--. 1 ceph ceph 14K May 10 14:28 /var/log/ceph/ceph-mds.ceph-24.log ceph-10: -rw-r--r--. 1 ceph ceph 394 May 10 14:02 /var/log/ceph/ceph-mds.ceph-10.log ceph-13: -rw-r--r--. 1 ceph ceph 394 May 10 14:02 /var/log/ceph/ceph-mds.ceph-13.log ceph-08: -rw-r--r--. 1 ceph ceph 394 May 10 14:02 /var/log/ceph/ceph-mds.ceph-08.log ceph-15: -rw-r--r--. 1 ceph ceph 14K May 10 14:28 /var/log/ceph/ceph-mds.ceph-15.log ceph-17: -rw-r--r--. 1 ceph ceph 14K May 10 14:28 /var/log/ceph/ceph-mds.ceph-17.log ceph-14: -rw-r--r--. 1 ceph ceph 16K May 10 14:28 /var/log/ceph/ceph-mds.ceph-14.log ceph-09: -rw-r--r--. 1 ceph ceph 16K May 10 14:28 /var/log/ceph/ceph-mds.ceph-09.log ceph-16: -rw-r--r--. 1 ceph ceph 15K May 10 14:28 /var/log/ceph/ceph-mds.ceph-16.log ceph-11: -rw-r--r--. 1 ceph ceph 14K May 10 14:28 /var/log/ceph/ceph-mds.ceph-11.log ceph-12: -rw-r--r--. 1 ceph ceph 394 May 10 14:02 /var/log/ceph/ceph-mds.ceph-12.log Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 24