- ceph-users - lists.ceph.io

Replace OSD node without remapping PGs
by Nghia Viet Tran 02 Apr '20

02 Apr '20

Hi everyone, I'm working on replacing OSDs node with the newer one. The new host has the new hostname and new disk (faster one but the same size with old disk). My plan is - Reweight the OSD to zero to spread all existed data to the rest nodes to keep data availability - set flag noout, norebalance, norecover, nobackfill, destroy the OSD and join the new OSD as the same ID of the old one. By above approach, the cluster will remap PGs of all nodes. Each data will be moved twice times until it reach the new OSD (reweight and join new node as same ID) I also did the other way that only set flags and destroy OSD. But the result is still the same (degraded objects from destroyed osd and misplaced object after joining new osd) Are there any ways to replace the OSD node directly without remapping PGs of the whole cluster? Many thanks! Nghia.

3 3

Using sendfile on Ceph FS results in data stuck in client cache
by Mikael Öhman 01 Apr '20

01 Apr '20

Hi all, Using sendfile function to write data to cephfs, the data doesn't end up being written. From the client that writes the file, it looks correct at first, but from all other ceph clients, the size is 0 bytes. Re-mounting the filesystem, the data is lost. I didn't see any errors, the data just doesn't get written, as if it's just cached in cephfs client. Writing just an extra byte at the end of the file (without sendfile), it seems to trigger the actual write of all the data. Could someone else confirm if they are also seeing such issue? I'm on ceph 13.2.8, using kernel module for mounting on CentOS7. I've used this sendfile-example for the example below: https://github.com/pijewski/sendfile-example/blob/master/sendfile.c Using a small 27 byte source file. # ls -lh examples/ -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src # ./sendfile examples/src examples/dst 27 # ls -lh examples/ ------x--- 1 root c3-staff 27 Mar 24 18:12 dst -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src But, directory is still on 27 bytes: # ls -lhd examples drwxr-sr-x 1 root c3-staff 27 Mar 24 18:15 examples and on all other cephfs clients, the file is empty: # ls -lh examples/ ------x--- 1 root c3-staff 0 Mar 24 18:12 dst -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src Is this a bug in cephfs, or should I not expect sendfile to work (as it is not posix compliant). There are no error reported from what i can see, and it is 100% reproducible Best regards, Mikael

2 4

Re: osd can not start at boot after upgrade to octopus
by Eugen Block 01 Apr '20

01 Apr '20

Resending the response back to the list. Zitat von "Lomayani S. Laizer" <lomlaizer(a)gmail.com>: > Hello, > I have been running Nautilus from May last year so this is separate issue > from recent bug > > I think the problem is between systemd and ceph-volume. No any logs hitting > osd logs because osd dont start at all. > starting osd manually works fine (/usr/bin/ceph-osd -f --cluster ceph --id > 29 --setuser ceph --setgroup ceph) > > You can see starting osd just exit with no usable log(RuntimeError: command > returned non-zero exit status: 1) > > Below is the logs of ceph-volume-systemd.log > Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir > --dev /dev/ceph-3811ddf5-02be-40f1-a53e-053131aa5712/osd-bl > ock-3e52d340-5416-46e6-b697-c15ca85f6883 --path /var/lib/ceph/osd/ceph-29 > --no-mon-config > Running command: /bin/ln -snf > /dev/ceph-3811ddf5-02be-40f1-a53e-053131aa5712/osd-block-3e52d340-5416-46e6-b697-c15ca85f6883 > /var/lib/c > eph/osd/ceph-29/block > Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-29/block > Running command: /bin/chown -R ceph:ceph /dev/dm-4 > Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-29 > Running command: /bin/systemctl enable > ceph-volume@lvm-29-3e52d340-5416-46e6-b697-c15ca85f6883 > Running command: /bin/systemctl enable --runtime ceph-osd@29 > stderr: Created symlink /run/systemd/system/ce > [2020-04-01 12:17:31,111][ceph_volume.process][INFO ] stderr > ph-osd.target.wants/ceph-osd(a)33.service → /lib/systemd/system/ceph-osd@. > service. > [2020-04-01 12:17:31,111][ceph_volume.process][INFO ] stderr > ph-osd.target.wants/ceph-osd(a)29.service → /lib/systemd/system/ceph-osd@. > service. > Running command: /bin/systemctl start ceph-osd@29 > stderr: Job for ceph-osd(a)29.service canceled. > --> RuntimeError: command returned non-zero exit status: 1 > > On Wed, Apr 1, 2020 at 1:22 PM Eugen Block <eblock(a)nde.ag> wrote: > >> Hi, >> >> are you hitting [1]? Did you run Nautilus only for a short period of >> time before upgrading to Octopus? >> >> If this doesn't apply to you, can you see anything in the OSD logs >> (/var/log/ceph/ceph-osd.<ID>.log)? >> >> Regards, >> Eugen >> >> [1] https://tracker.ceph.com/issues/44770 >> >> Zitat von "Lomayani S. Laizer" <lomlaizer(a)gmail.com>: >> >> > Hello, >> > I have upgraded nautilus cluster to octopus few days ago. the cluster was >> > running ok and even after to octopus everything was running ok >> > >> > the issue came when i rebooted the servers for updating the kernel. Two >> > servers out of 6 osd's servers osd cant start. No error reported in >> > ceph-volume.log and ceph-volume-systemd.log >> > >> > Starting osd with /usr/bin/ceph-osd -f --cluster ceph --id 30 --setuser >> > ceph --setgroup ceph works just fine. the issue is starting osd in >> systemd >> > >> > ceph-volume-systemd.log >> > >> > 16:36:28,193][systemd][WARNING] failed activating OSD, retries left: 30 >> > [2020-03-31 16:36:28,196][systemd][WARNING] command returned non-zero >> exit >> > status: 1 >> > [2020-03-31 16:36:28,196][systemd][WARNING] failed activating OSD, >> retries >> > left: 30 >> > [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: >> > lvm-28-7f4113c8-c5cf-4f70-9f7a-7a32de9d6587 >> > [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: >> > lvm-30-8a70ad95-1c79-4502-a9a3-d5d7b9df84b6 >> > [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: >> > lvm-31-a8efb7db-686b-4789-a9c4-01442c28577f >> > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra >> > data: 28-7f4113c8-c5cf-4f70-9f7a-7a32de9d6587 >> > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra >> > data: 30-8a70ad95-1c79-4502-a9a3-d5d7b9df84b6 >> > [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: >> > lvm-33-7d688fc1-ed7b-45ae-ac0e-7b1787e0b64f >> > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra >> > data: 31-a8efb7db-686b-4789-a9c4-01442c28577f >> > [2020-03-31 16:41:25,068][systemd][INFO ] raw systemd input received: >> > lvm-29-3e52d340-5416-46e6-b697-c15ca85f6883 >> > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra >> > data: 33-7d688fc1-ed7b-45ae-ac0e-7b1787e0b64f >> > [2020-03-31 16:41:25,068][systemd][INFO ] raw systemd input received: >> > lvm-32-3841a62d-d6bc-404a-8762-163530b2d5d4 >> > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra >> > data: 29-3e52d340-5416-46e6-b697-c15ca85f6883 >> > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra >> > data: 32-3841a62d-d6bc-404a-8762-163530b2d5d4 >> > [2020-03-31 16:41:25,108][ceph_volume.process][INFO ] Running command: >> > /usr/sbin/ceph-volume lvm trigger 29-3e52d340-5416-46e6-b697-c15ca85f6883 >> > >> > ceph-volume.log >> > 2-163530b2d5d4 >> > [2020-03-31 17:17:23,679][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl enable --runtime ceph-osd@31 >> > [2020-03-31 17:17:23,863][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl enable --runtime ceph-osd@30 >> > [2020-03-31 17:17:24,045][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl enable --runtime ceph-osd@33 >> > [2020-03-31 17:17:24,241][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl enable --runtime ceph-osd@32 >> > [2020-03-31 17:17:24,449][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl enable --runtime ceph-osd@28 >> > [2020-03-31 17:17:24,629][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl enable --runtime ceph-osd@29 >> > [2020-03-31 17:17:24,652][ceph_volume.process][INFO ] stderr Created >> > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)31.service → >> > /lib/systemd/system/ceph-osd@.service. >> > [2020-03-31 17:17:24,664][ceph_volume.process][INFO ] stderr Created >> > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)30.service → >> > /lib/systemd/system/ceph-osd@.service. >> > [2020-03-31 17:17:24,872][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl start ceph-osd@31 >> > [2020-03-31 17:17:24,875][ceph_volume.process][INFO ] stderr Created >> > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)33.service → >> > /lib/systemd/system/ceph-osd@.service. >> > [2020-03-31 17:17:25,072][ceph_volume.process][INFO ] stderr Created >> > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)32.service → >> > /lib/systemd/system/ceph-osd@.service. >> > [2020-03-31 17:17:25,075][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl start ceph-osd@30 >> > [2020-03-31 17:17:25,282][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl start ceph-osd@33 >> > [2020-03-31 17:17:25,497][ceph_volume.process][INFO ] stderr Created >> > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)28.service → >> > /lib/systemd/system/ceph-osd@.service. >> > [2020-03-31 17:17:25,499][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl start ceph-osd@32 >> > [2020-03-31 17:17:25,520][ceph_volume.process][INFO ] stderr Created >> > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)29.service → >> > /lib/systemd/system/ceph-osd@.service. >> > [2020-03-31 17:17:25,705][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl start ceph-osd@28 >> > [2020-03-31 17:17:25,887][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl start ceph-osd@29 >> > >> > -- >> > Lomayani >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

1 0

Re: [Octopus] Beware the on-disk conversion
by Dan van der Ster 01 Apr '20

01 Apr '20

Doh, I hope so! On Wed, Apr 1, 2020 at 5:35 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > April fools day!!!!!! :) > > > -----Original Message----- > Sent: 01 April 2020 17:28 > To: ceph-users(a)ceph.io > Subject: [ceph-users] [Octopus] Beware the on-disk conversion > > Hi, > > As the upgrade documentation tells: > > Note that the first time each OSD starts, it will do a format > > conversion to improve the accounting for omap data. This may take > a > > few minutes to as much as a few hours (for an HDD with lots of omap > > data). You can disable this automatic conversion with: > > What the documentation does not say is that this process takes a lot of > memory > > I am upgrading a rusty cluster from Nautilus, you can check out the ram > consumption as attachment > > First, we have a 3TB osd conversion: it tooks ~15min, and 19GB of memory > > Then, we have a larger 6TB osd conversion: it tooks more than 2 hours, > and 35GB of memory > > Finally, you have the largest 10TB osd: only 1H15, but 52GB of memory > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Bluestore compression parameters in ceph.conf not used in mimic 13.2.8?
by Frank Schilder 01 Apr '20

01 Apr '20

Dear all, I have two observations regarding bluestore compression config: 1) ceph.conf settings seem to be ignored. 2) The SSD default values seem not to save space using compression. To 1) We are running a mimic 13.2.8 cluster with OSDs deployed under mimic 13.2.2. Back then the interpretation of compression parameters was messed up, which has been fixed along the way from 13.2.2 to 13.2.8. To get it to work properly under 13.2.2 I needed to include these settings in ceph.conf: [osd] bluestore compression mode = aggressive bluestore compression min blob size hdd = 262144 and then also enable compression on all pools that should use compression. These settings are still present in ceph.conf, but they seem to be ignored when populating the config data base on mon startup or querying config parameters: # ceph config get osd.16 bluestore_compression_min_blob_size_hdd 131072 However: # ceph tell osd.16 config get bluestore_compression_min_blob_size_hdd 262144 and: # ceph config show osd.16 NAME VALUE SOURCE OVERRIDES IGNORES bluestore_compression_min_blob_size_hdd 262144 file This is really confusing. Is this intended? Which values will be used when deploying new OSDs? In general, it would really be helpful if one could query daemon/parameter groups as in " ceph config get osd bluestore_compression_min_blob_size_hdd" to get a list right away. To 2) In a long-long-ago discussion about how compression works, I was told that a blob of bluestore_compression_min_blob_size will be compressed and then distributed over a number of allocations of bluestore_min_alloc_size. The defaults for HDD and SSD are: bluestore_compression_min_blob_size_hdd 131072 bluestore_min_alloc_size_hdd 65536 bluestore_compression_min_blob_size_ssd 8192 bluestore_min_alloc_size_ssd 16384 If this explanation of the compression method is correct, these defaults allow up to 50% savings for HDD, but, erm, 0% for SSD as the uncompressed blob will use the same amount of space as the compressed one as both will require the same allocation size. Did something change here? Are compressed blobs now co-located in allocations? Thanks for your help, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

osd can not start at boot after upgrade to octopus
by Lomayani S. Laizer 01 Apr '20

01 Apr '20

Hello, I have upgraded nautilus cluster to octopus few days ago. the cluster was running ok and even after to octopus everything was running ok the issue came when i rebooted the servers for updating the kernel. Two servers out of 6 osd's servers osd cant start. No error reported in ceph-volume.log and ceph-volume-systemd.log Starting osd with /usr/bin/ceph-osd -f --cluster ceph --id 30 --setuser ceph --setgroup ceph works just fine. the issue is starting osd in systemd ceph-volume-systemd.log 16:36:28,193][systemd][WARNING] failed activating OSD, retries left: 30 [2020-03-31 16:36:28,196][systemd][WARNING] command returned non-zero exit status: 1 [2020-03-31 16:36:28,196][systemd][WARNING] failed activating OSD, retries left: 30 [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: lvm-28-7f4113c8-c5cf-4f70-9f7a-7a32de9d6587 [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: lvm-30-8a70ad95-1c79-4502-a9a3-d5d7b9df84b6 [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: lvm-31-a8efb7db-686b-4789-a9c4-01442c28577f [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra data: 28-7f4113c8-c5cf-4f70-9f7a-7a32de9d6587 [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra data: 30-8a70ad95-1c79-4502-a9a3-d5d7b9df84b6 [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: lvm-33-7d688fc1-ed7b-45ae-ac0e-7b1787e0b64f [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra data: 31-a8efb7db-686b-4789-a9c4-01442c28577f [2020-03-31 16:41:25,068][systemd][INFO ] raw systemd input received: lvm-29-3e52d340-5416-46e6-b697-c15ca85f6883 [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra data: 33-7d688fc1-ed7b-45ae-ac0e-7b1787e0b64f [2020-03-31 16:41:25,068][systemd][INFO ] raw systemd input received: lvm-32-3841a62d-d6bc-404a-8762-163530b2d5d4 [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra data: 29-3e52d340-5416-46e6-b697-c15ca85f6883 [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra data: 32-3841a62d-d6bc-404a-8762-163530b2d5d4 [2020-03-31 16:41:25,108][ceph_volume.process][INFO ] Running command: /usr/sbin/ceph-volume lvm trigger 29-3e52d340-5416-46e6-b697-c15ca85f6883 ceph-volume.log 2-163530b2d5d4 [2020-03-31 17:17:23,679][ceph_volume.process][INFO ] Running command: /bin/systemctl enable --runtime ceph-osd@31 [2020-03-31 17:17:23,863][ceph_volume.process][INFO ] Running command: /bin/systemctl enable --runtime ceph-osd@30 [2020-03-31 17:17:24,045][ceph_volume.process][INFO ] Running command: /bin/systemctl enable --runtime ceph-osd@33 [2020-03-31 17:17:24,241][ceph_volume.process][INFO ] Running command: /bin/systemctl enable --runtime ceph-osd@32 [2020-03-31 17:17:24,449][ceph_volume.process][INFO ] Running command: /bin/systemctl enable --runtime ceph-osd@28 [2020-03-31 17:17:24,629][ceph_volume.process][INFO ] Running command: /bin/systemctl enable --runtime ceph-osd@29 [2020-03-31 17:17:24,652][ceph_volume.process][INFO ] stderr Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)31.service → /lib/systemd/system/ceph-osd@.service. [2020-03-31 17:17:24,664][ceph_volume.process][INFO ] stderr Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)30.service → /lib/systemd/system/ceph-osd@.service. [2020-03-31 17:17:24,872][ceph_volume.process][INFO ] Running command: /bin/systemctl start ceph-osd@31 [2020-03-31 17:17:24,875][ceph_volume.process][INFO ] stderr Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)33.service → /lib/systemd/system/ceph-osd@.service. [2020-03-31 17:17:25,072][ceph_volume.process][INFO ] stderr Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)32.service → /lib/systemd/system/ceph-osd@.service. [2020-03-31 17:17:25,075][ceph_volume.process][INFO ] Running command: /bin/systemctl start ceph-osd@30 [2020-03-31 17:17:25,282][ceph_volume.process][INFO ] Running command: /bin/systemctl start ceph-osd@33 [2020-03-31 17:17:25,497][ceph_volume.process][INFO ] stderr Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)28.service → /lib/systemd/system/ceph-osd@.service. [2020-03-31 17:17:25,499][ceph_volume.process][INFO ] Running command: /bin/systemctl start ceph-osd@32 [2020-03-31 17:17:25,520][ceph_volume.process][INFO ] stderr Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)29.service → /lib/systemd/system/ceph-osd@.service. [2020-03-31 17:17:25,705][ceph_volume.process][INFO ] Running command: /bin/systemctl start ceph-osd@28 [2020-03-31 17:17:25,887][ceph_volume.process][INFO ] Running command: /bin/systemctl start ceph-osd@29 -- Lomayani

2 2

No reply or very slow reply from Prometheus plugin - ceph-mgr 13.2.8 mimic
by Paul Choi 01 Apr '20

01 Apr '20

Hello, We are running Mimic 13.2.8 with our cluster, and since upgrading to 13.2.8 the Prometheus plugin seems to hang a lot. It used to respond under 10s but now it often hangs. Restarting the mgr processes helps temporarily but within minutes it gets stuck again. The active mgr doesn't exit when doing `systemctl stop ceph-mgr.target" and needs to be kill -9'ed. Is there anything I can do to address this issue, or at least get better visibility into the issue? We only have a few plugins enabled: $ ceph mgr module ls { "enabled_modules": [ "balancer", "prometheus", "zabbix" ], 3 mgr processes, but it's a pretty large cluster (near 4000 OSDs) and it's a busy one with lots of rebalancing. (I don't know if a busy cluster would seriously affect the mgr's performance, but just throwing it out there) services: mon: 5 daemons, quorum woodenbox0,woodenbox2,woodenbox4,woodenbox3,woodenbox1 mgr: woodenbox2(active), standbys: woodenbox0, woodenbox1 mds: cephfs-1/1/1 up {0=woodenbox6=up:active}, 1 up:standby-replay osd: 3964 osds: 3928 up, 3928 in; 831 remapped pgs rgw: 4 daemons active Thanks in advance for your help, -Paul Choi

4 13

Multiple CephFS creation
by Jarett DeAngelis 01 Apr '20

01 Apr '20

Hi guys, This is documented as an experimental feature, but it doesn’t explain how to ensure that affinity for a given MDS sticks to the second filesystem you create. Has anyone had success implementing a second CephFS? In my case it will be based on a completely different pool from my first one. Thanks. J

4 8

Re: Questions on Ceph cluster without OS disks
by Eric Petit 01 Apr '20

01 Apr '20

It works well for me, been running a couple clusters for 1-2 years where all OSD hosts (~200) have no system disks and instead netboot from PXE. No NFS server involved, each host loads the same system image (Debian Live squashfs) into memory on boot and runs independently from there on out. Takes some trickery to configure and bring the OSDs up on boot (using puppet in my case), though that might get easier with the containerized approach in Ceph 15+. Best, Eric > On 21 Mar 2020, at 14:18, huxiaoyu(a)horebdata.cn wrote: > > Hi, Marc, > > Indeed PXE boot makes a lot sense in large cluster, cuting down OS deployment and management burden, but only iff no single of failure is guaranteed... > > best regards, > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: Marc Roos > Date: 2020-03-21 14:13 > To: ceph-users; huxiaoyu; martin.verges > Subject: RE: [ceph-users] Questions on Ceph cluster without OS disks > > I would say it is not a 'proven technology' otherwise you would see a > wide spread implementation and adaptation of this method. However if you > really need the physical disk space, it is a solution. Although I also > would have questions on creating an extra redundant environment to > service remote booting, just to spare a os disk position. Maybe this > makes more sence in really big environments. > > > > > > -----Original Message----- > From: huxiaoyu(a)horebdata.cn [mailto:huxiaoyu@horebdata.cn] > Sent: 21 March 2020 13:54 > To: Martin Verges; ceph-users > Subject: [ceph-users] Questions on Ceph cluster without OS disks > > Hello， Martin， > > I notice that Croit advocate the use of ceph cluster without OS disks, > but with PXE boot. > > Do you use a NFS server to serve the root file system for each node? > such as hosting configuration files, user and password, log files, etc. > My question is, will the NFS server be a single point of failure? If the > NFS server goes down, the network experience any outage, ceph nodes may > not be able to write to the local file systems, possibly leading to > service outage. > > How do you deal with the above potential issues in production? I am a > bit worried... > > best regards, > > samuel > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph WAL/DB disks - do they really only use 3GB, or 30Gb, or 300GB
by victorhooi＠yahoo.com 01 Apr '20

01 Apr '20

Hi, I'm using Intel Optane disks to provide WAL/DB capacity for my Ceph cluster (which is part of Proxmox - for VM hosting). I've read that WAL/DB partitions only use either 3GB, or 30GB, or 300GB - due to the way that RocksDB works. Is this true? My current partition for WAL/DB is 145 GB - does this mean that 115Gb of that will be permanently wasted? Is this behaviour documented somewhere, or is there some background, so I can understand a bit more about how it works? Thanks, Victor

2 1