hmm ... I tried a: * laptop restart too, to do away with stale processes * removed build directory and did a full rebuild * removed the out/ and dev/ directories But none of that seems to work. I wonder if this is anything related to selinux labeling on my root or /home file-system. Here's the vstart output: $ MDS=3 ../src/vstart.sh -n --without-dashboard rm -f core* hostname indraprastha ip 192.168.1.135 port 40296 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-authtool --create-keyring --gen-key --name=mon. /home/mchangir/work/mchangir-ceph.git/build/keyring --cap mon 'allow *' creating /home/mchangir/work/mchangir-ceph.git/build/keyring /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /home/mchangir/work/mchangir-ceph.git/build/keyring /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-authtool --gen-key --name=client.fs --cap mon 'allow r' --cap osd 'allow rw tag cephfs data=*' --cap mds 'allow rwp' /home/mchangir/work/mchangir-ceph.git/build/keyring /home/mchangir/work/mchangir-ceph.git/build/bin/monmaptool --create --clobber --addv a [v2:192.168.1.135:40296,v1:192.168.1.135:40297] --addv b [v2:192.168.1.135:40298,v1:192.168.1.135:40299] --addv c [v2: 192.168.1.135:40300,v1:192.168.1.135:40301] --print /tmp/ceph_monmap.1043 /home/mchangir/work/mchangir-ceph.git/build/bin/monmaptool: monmap file /tmp/ceph_monmap.1043 /home/mchangir/work/mchangir-ceph.git/build/bin/monmaptool: generated fsid 49f79832-fc87-40e1-89f1-be27391c4971 epoch 0 fsid 49f79832-fc87-40e1-89f1-be27391c4971 last_changed 2019-10-25T11:04:33.175225+0530 created 2019-10-25T11:04:33.175225+0530 min_mon_release 0 (unknown) 0: [v2:192.168.1.135:40296/0,v1:192.168.1.135:40297/0] mon.a 1: [v2:192.168.1.135:40298/0,v1:192.168.1.135:40299/0] mon.b 2: [v2:192.168.1.135:40300/0,v1:192.168.1.135:40301/0] mon.c /home/mchangir/work/mchangir-ceph.git/build/bin/monmaptool: writing epoch 0 to /tmp/ceph_monmap.1043 (3 monitors) rm -rf -- /home/mchangir/work/mchangir-ceph.git/build/dev/mon.a mkdir -p /home/mchangir/work/mchangir-ceph.git/build/dev/mon.a /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon --mkfs -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -i a --monmap=/tmp/ceph_monmap.1043 --keyring=/home/mchangir/work/mchangir-ceph.git/build/keyring rm -rf -- /home/mchangir/work/mchangir-ceph.git/build/dev/mon.b mkdir -p /home/mchangir/work/mchangir-ceph.git/build/dev/mon.b /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon --mkfs -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -i b --monmap=/tmp/ceph_monmap.1043 --keyring=/home/mchangir/work/mchangir-ceph.git/build/keyring rm -rf -- /home/mchangir/work/mchangir-ceph.git/build/dev/mon.c mkdir -p /home/mchangir/work/mchangir-ceph.git/build/dev/mon.c /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon --mkfs -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -i c --monmap=/tmp/ceph_monmap.1043 --keyring=/home/mchangir/work/mchangir-ceph.git/build/keyring rm -- /tmp/ceph_monmap.1043 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon -i a -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon -i b -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon -i c -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf Populating config ... [mgr] mgr/telemetry/enable = false mgr/telemetry/nag = false creating /home/mchangir/work/mchangir-ceph.git/build/dev/mgr.x/keyring /home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring -i /home/mchangir/work/mchangir-ceph.git/build/dev/mgr.x/keyring auth add mgr.x mon 'allow profile mgr' mds 'allow *' osd 'allow *' added key for mgr.x /home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring config set mgr mgr/restful/x/server_port 42296 --force Starting mgr.x /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mgr -i x -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf waiting for mgr restful module to start waiting for mgr restful module to start waiting for mgr restful module to start waiting for mgr restful module to start /home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring restful create-self-signed-cert Restarting RESTful API server... /home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring restful create-key admin -o /tmp/tmp.8g3nFGa8Tt add osd0 f9e8807c-eb74-49d4-a6ee-b1e2a163d903 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring osd new f9e8807c-eb74-49d4-a6ee-b1e2a163d903 -i /home/mchangir/work/mchangir-ceph.git/build/dev/osd0/new.json 0 2019-10-25T11:04:47.258+0530 7f83ea141e00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd0/keyring: (2) No such file or directory 2019-10-25T11:04:47.259+0530 7f83ea141e00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd0/keyring: (2) No such file or directory 2019-10-25T11:04:47.259+0530 7f83ea141e00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd0/keyring: (2) No such file or directory 2019-10-25T11:04:47.268+0530 7f83ea141e00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd0/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd0/block: (2) No such file or directory 2019-10-25T11:04:47.268+0530 7f83ea141e00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd0/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd0/block: (2) No such file or directory 2019-10-25T11:04:47.268+0530 7f83ea141e00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd0/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd0/block: (2) No such file or directory 2019-10-25T11:04:47.270+0530 7f83ea141e00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd0) _read_fsid unparsable uuid start osd.0 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-osd -i 0 -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf 2019-10-25T11:04:59.565+0530 7f466be82e00 -1 Falling back to public interface 2019-10-25T11:05:05.290+0530 7f466be82e00 -1 osd.0 0 log_to_monitors {default=true} add osd1 0382fdae-2bb9-43cc-b356-6546c1d4d7a4 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring osd new 0382fdae-2bb9-43cc-b356-6546c1d4d7a4 -i /home/mchangir/work/mchangir-ceph.git/build/dev/osd1/new.json 1 2019-10-25T11:05:06.868+0530 7f28cd0efe00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd1/keyring: (2) No such file or directory 2019-10-25T11:05:06.869+0530 7f28cd0efe00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd1/keyring: (2) No such file or directory 2019-10-25T11:05:06.869+0530 7f28cd0efe00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd1/keyring: (2) No such file or directory 2019-10-25T11:05:06.877+0530 7f28cd0efe00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd1/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd1/block: (2) No such file or directory 2019-10-25T11:05:06.878+0530 7f28cd0efe00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd1/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd1/block: (2) No such file or directory 2019-10-25T11:05:06.878+0530 7f28cd0efe00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd1/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd1/block: (2) No such file or directory 2019-10-25T11:05:06.879+0530 7f28cd0efe00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd1) _read_fsid unparsable uuid start osd.1 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-osd -i 1 -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf 2019-10-25T11:05:19.258+0530 7ff59398ae00 -1 Falling back to public interface 2019-10-25T11:05:24.742+0530 7ff59398ae00 -1 osd.1 0 log_to_monitors {default=true} add osd2 c1ef31df-7adf-4039-8640-b8b193b8a84b /home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring osd new c1ef31df-7adf-4039-8640-b8b193b8a84b -i /home/mchangir/work/mchangir-ceph.git/build/dev/osd2/new.json 2 2019-10-25T11:05:26.574+0530 7f716604be00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd2/keyring: (2) No such file or directory 2019-10-25T11:05:26.575+0530 7f716604be00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd2/keyring: (2) No such file or directory 2019-10-25T11:05:26.575+0530 7f716604be00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd2/keyring: (2) No such file or directory 2019-10-25T11:05:26.583+0530 7f716604be00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd2/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd2/block: (2) No such file or directory 2019-10-25T11:05:26.583+0530 7f716604be00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd2/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd2/block: (2) No such file or directory 2019-10-25T11:05:26.583+0530 7f716604be00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd2/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd2/block: (2) No such file or directory 2019-10-25T11:05:26.585+0530 7f716604be00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd2) _read_fsid unparsable uuid start osd.2 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-osd -i 2 -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf 2019-10-25T11:05:38.890+0530 7f284ee18e00 -1 Falling back to public interface 2019-10-25T11:05:44.365+0530 7f284ee18e00 -1 osd.2 0 log_to_monitors {default=true} mkdir -p /home/mchangir/work/mchangir-ceph.git/build/dev/mds.a /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-authtool --create-keyring --gen-key --name=mds.a /home/mchangir/work/mchangir-ceph.git/build/dev/mds.a/keyring creating /home/mchangir/work/mchangir-ceph.git/build/dev/mds.a/keyring /home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring -i /home/mchangir/work/mchangir-ceph.git/build/dev/mds.a/keyring auth add mds.a mon 'allow profile mds' osd 'allow rw tag cephfs *=*' mds allow mgr 'allow profile mds' added key for mds.a /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mds -i a -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf starting mds.a at mkdir -p /home/mchangir/work/mchangir-ceph.git/build/dev/mds.b /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-authtool --create-keyring --gen-key --name=mds.b /home/mchangir/work/mchangir-ceph.git/build/dev/mds.b/keyring creating /home/mchangir/work/mchangir-ceph.git/build/dev/mds.b/keyring /home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring -i /home/mchangir/work/mchangir-ceph.git/build/dev/mds.b/keyring auth add mds.b mon 'allow profile mds' osd 'allow rw tag cephfs *=*' mds allow mgr 'allow profile mds' added key for mds.b /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mds -i b -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf starting mds.b at mkdir -p /home/mchangir/work/mchangir-ceph.git/build/dev/mds.c /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-authtool --create-keyring --gen-key --name=mds.c /home/mchangir/work/mchangir-ceph.git/build/dev/mds.c/keyring creating /home/mchangir/work/mchangir-ceph.git/build/dev/mds.c/keyring /home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring -i /home/mchangir/work/mchangir-ceph.git/build/dev/mds.c/keyring auth add mds.c mon 'allow profile mds' osd 'allow rw tag cephfs *=*' mds allow mgr 'allow profile mds' added key for mds.c /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mds -i c -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf starting mds.c at /home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring fs volume ls [] /home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring fs volume create a Volume created successfully (no MDS daemons created) /home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring fs authorize a client.fs_a / rwp [client.fs_a] key = AQC7ibJd4WUhIhAAzQS3FmIc2bIgpNEwHXqyhg== started. stop.sh to stop. see out/* (e.g. 'tail -f out/????') for debug output. restful urls: https://192.168.1.135:42296 w/ user/pass: admin / 743f9239-ca11-43ca-a286-c88b76c86720 ---------- Here's the ceph processes listing: $ ps aux | grep 'ceph-' mchangir 1216 4.0 0.5 461008 88592 ? Ssl 11:04 0:03 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon -i a -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf mchangir 1320 2.6 0.5 459988 82828 ? Ssl 11:04 0:02 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon -i b -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf mchangir 1375 3.3 0.5 459984 88740 ? Ssl 11:04 0:03 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon -i c -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf mchangir 1530 3.6 1.6 1020260 270964 ? Ssl 11:04 0:03 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mgr -i x -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf mchangir 1994 6.0 0.6 917136 111156 ? Ssl 11:04 0:04 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-osd -i 0 -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf mchangir 2545 8.5 0.7 917140 114748 ? Ssl 11:05 0:04 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-osd -i 1 -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf mchangir 3138 13.0 0.7 917148 113052 ? Ssl 11:05 0:04 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-osd -i 2 -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf mchangir 3625 0.5 0.2 403724 34416 pts/1 Sl 11:05 0:00 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mds -i a -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf mchangir 3742 0.2 0.1 337136 25476 pts/1 Sl 11:05 0:00 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mds -i b -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf mchangir 3800 0.2 0.1 338160 26368 pts/1 Sl 11:05 0:00 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mds -i c -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf On Fri, Oct 25, 2019 at 4:36 AM Patrick Donnelly <pdonnell@redhat.com> wrote:

On Mon, Oct 21, 2019 at 6:59 AM Milind Changire <mchangir@redhat.com> wrote:

...

after running vstart cluster on my laptop, all ./bin/ceph commands are

...

I can also see that ceph-osd processes are still attached to the tty; is

hung; this expected ?

This could be anything. You need to share output from vstart. I recommend removing the build directory and rebuilding first if you haven't already. Make sure there are no lost ceph daemons on your system too.

-- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

-- Milind

On Mon, Oct 28, 2019 at 11:23 PM Patrick Donnelly <pdonnell@redhat.com> wrote:

On Thu, Oct 24, 2019 at 11:05 PM Milind Changire <mchangir@redhat.com> wrote:

...

hmm ... I tried a: * laptop restart too, to do away with stale processes * removed build directory and did a full rebuild * removed the out/ and dev/ directories But none of that seems to work. I wonder if this is anything related to selinux labeling on my root or

/home file-system.

...

Here's the vstart output: $ MDS=3 ../src/vstart.sh -n --without-dashboard [...]

Nothing wrong with the vstart output I see. Can you share `ceph status` and which command is hanging?

`ceph status` command hangs as well The command I was attempting was: `ceph osd pool ls`