hmm ... I tried a:
* laptop restart too, to do away with stale processes
* removed build directory and did a full rebuild
* removed the out/ and dev/ directories
But none of that seems to work.
I wonder if this is anything related to selinux labeling on my root or /home file-system.
Here's the vstart output:
$ MDS=3 ../src/vstart.sh -n --without-dashboard
rm -f core*
hostname indraprastha
ip 192.168.1.135
port 40296
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-authtool --create-keyring --gen-key --name=mon. /home/mchangir/work/mchangir-ceph.git/build/keyring --cap mon 'allow *'
creating /home/mchangir/work/mchangir-ceph.git/build/keyring
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /home/mchangir/work/mchangir-ceph.git/build/keyring
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-authtool --gen-key --name=client.fs --cap mon 'allow r' --cap osd 'allow rw tag cephfs data=*' --cap mds 'allow rwp' /home/mchangir/work/mchangir-ceph.git/build/keyring
/home/mchangir/work/mchangir-ceph.git/build/bin/monmaptool --create --clobber --addv a [v2:192.168.1.135:40296,v1:192.168.1.135:40297] --addv b [v2:192.168.1.135:40298,v1:192.168.1.135:40299] --addv c [v2:192.168.1.135:40300,v1:192.168.1.135:40301] --print /tmp/ceph_monmap.1043
/home/mchangir/work/mchangir-ceph.git/build/bin/monmaptool: monmap file /tmp/ceph_monmap.1043
/home/mchangir/work/mchangir-ceph.git/build/bin/monmaptool: generated fsid 49f79832-fc87-40e1-89f1-be27391c4971
epoch 0
fsid 49f79832-fc87-40e1-89f1-be27391c4971
last_changed 2019-10-25T11:04:33.175225+0530
created 2019-10-25T11:04:33.175225+0530
min_mon_release 0 (unknown)
0: [v2:192.168.1.135:40296/0,v1:192.168.1.135:40297/0] mon.a
1: [v2:192.168.1.135:40298/0,v1:192.168.1.135:40299/0] mon.b
2: [v2:192.168.1.135:40300/0,v1:192.168.1.135:40301/0] mon.c
/home/mchangir/work/mchangir-ceph.git/build/bin/monmaptool: writing epoch 0 to /tmp/ceph_monmap.1043 (3 monitors)
rm -rf -- /home/mchangir/work/mchangir-ceph.git/build/dev/mon.a
mkdir -p /home/mchangir/work/mchangir-ceph.git/build/dev/mon.a
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon --mkfs -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -i a --monmap=/tmp/ceph_monmap.1043 --keyring=/home/mchangir/work/mchangir-ceph.git/build/keyring
rm -rf -- /home/mchangir/work/mchangir-ceph.git/build/dev/mon.b
mkdir -p /home/mchangir/work/mchangir-ceph.git/build/dev/mon.b
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon --mkfs -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -i b --monmap=/tmp/ceph_monmap.1043 --keyring=/home/mchangir/work/mchangir-ceph.git/build/keyring
rm -rf -- /home/mchangir/work/mchangir-ceph.git/build/dev/mon.c
mkdir -p /home/mchangir/work/mchangir-ceph.git/build/dev/mon.c
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon --mkfs -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -i c --monmap=/tmp/ceph_monmap.1043 --keyring=/home/mchangir/work/mchangir-ceph.git/build/keyring
rm -- /tmp/ceph_monmap.1043
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon -i a -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon -i b -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon -i c -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
Populating config ...
[mgr]
mgr/telemetry/enable = false
mgr/telemetry/nag = false
creating /home/mchangir/work/mchangir-ceph.git/build/dev/mgr.x/keyring
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring -i /home/mchangir/work/mchangir-ceph.git/build/dev/mgr.x/keyring auth add mgr.x mon 'allow profile mgr' mds 'allow *' osd 'allow *'
added key for mgr.x
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring config set mgr mgr/restful/x/server_port 42296 --force
Starting mgr.x
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mgr -i x -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
waiting for mgr restful module to start
waiting for mgr restful module to start
waiting for mgr restful module to start
waiting for mgr restful module to start
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring restful create-self-signed-cert
Restarting RESTful API server...
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring restful create-key admin -o /tmp/tmp.8g3nFGa8Tt
add osd0 f9e8807c-eb74-49d4-a6ee-b1e2a163d903
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring osd new f9e8807c-eb74-49d4-a6ee-b1e2a163d903 -i /home/mchangir/work/mchangir-ceph.git/build/dev/osd0/new.json
0
2019-10-25T11:04:47.258+0530 7f83ea141e00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd0/keyring: (2) No such file or directory
2019-10-25T11:04:47.259+0530 7f83ea141e00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd0/keyring: (2) No such file or directory
2019-10-25T11:04:47.259+0530 7f83ea141e00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd0/keyring: (2) No such file or directory
2019-10-25T11:04:47.268+0530 7f83ea141e00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd0/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd0/block: (2) No such file or directory
2019-10-25T11:04:47.268+0530 7f83ea141e00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd0/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd0/block: (2) No such file or directory
2019-10-25T11:04:47.268+0530 7f83ea141e00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd0/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd0/block: (2) No such file or directory
2019-10-25T11:04:47.270+0530 7f83ea141e00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd0) _read_fsid unparsable uuid
start osd.0
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-osd -i 0 -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
2019-10-25T11:04:59.565+0530 7f466be82e00 -1 Falling back to public interface
2019-10-25T11:05:05.290+0530 7f466be82e00 -1 osd.0 0 log_to_monitors {default=true}
add osd1 0382fdae-2bb9-43cc-b356-6546c1d4d7a4
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring osd new 0382fdae-2bb9-43cc-b356-6546c1d4d7a4 -i /home/mchangir/work/mchangir-ceph.git/build/dev/osd1/new.json
1
2019-10-25T11:05:06.868+0530 7f28cd0efe00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd1/keyring: (2) No such file or directory
2019-10-25T11:05:06.869+0530 7f28cd0efe00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd1/keyring: (2) No such file or directory
2019-10-25T11:05:06.869+0530 7f28cd0efe00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd1/keyring: (2) No such file or directory
2019-10-25T11:05:06.877+0530 7f28cd0efe00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd1/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd1/block: (2) No such file or directory
2019-10-25T11:05:06.878+0530 7f28cd0efe00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd1/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd1/block: (2) No such file or directory
2019-10-25T11:05:06.878+0530 7f28cd0efe00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd1/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd1/block: (2) No such file or directory
2019-10-25T11:05:06.879+0530 7f28cd0efe00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd1) _read_fsid unparsable uuid
start osd.1
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-osd -i 1 -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
2019-10-25T11:05:19.258+0530 7ff59398ae00 -1 Falling back to public interface
2019-10-25T11:05:24.742+0530 7ff59398ae00 -1 osd.1 0 log_to_monitors {default=true}
add osd2 c1ef31df-7adf-4039-8640-b8b193b8a84b
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring osd new c1ef31df-7adf-4039-8640-b8b193b8a84b -i /home/mchangir/work/mchangir-ceph.git/build/dev/osd2/new.json
2
2019-10-25T11:05:26.574+0530 7f716604be00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd2/keyring: (2) No such file or directory
2019-10-25T11:05:26.575+0530 7f716604be00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd2/keyring: (2) No such file or directory
2019-10-25T11:05:26.575+0530 7f716604be00 -1 auth: unable to find a keyring on /home/mchangir/work/mchangir-ceph.git/build/dev/osd2/keyring: (2) No such file or directory
2019-10-25T11:05:26.583+0530 7f716604be00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd2/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd2/block: (2) No such file or directory
2019-10-25T11:05:26.583+0530 7f716604be00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd2/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd2/block: (2) No such file or directory
2019-10-25T11:05:26.583+0530 7f716604be00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd2/block) _read_bdev_label failed to open /home/mchangir/work/mchangir-ceph.git/build/dev/osd2/block: (2) No such file or directory
2019-10-25T11:05:26.585+0530 7f716604be00 -1 bluestore(/home/mchangir/work/mchangir-ceph.git/build/dev/osd2) _read_fsid unparsable uuid
start osd.2
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-osd -i 2 -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
2019-10-25T11:05:38.890+0530 7f284ee18e00 -1 Falling back to public interface
2019-10-25T11:05:44.365+0530 7f284ee18e00 -1 osd.2 0 log_to_monitors {default=true}
mkdir -p /home/mchangir/work/mchangir-ceph.git/build/dev/mds.a
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-authtool --create-keyring --gen-key --name=mds.a /home/mchangir/work/mchangir-ceph.git/build/dev/mds.a/keyring
creating /home/mchangir/work/mchangir-ceph.git/build/dev/mds.a/keyring
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring -i /home/mchangir/work/mchangir-ceph.git/build/dev/mds.a/keyring auth add mds.a mon 'allow profile mds' osd 'allow rw tag cephfs *=*' mds allow mgr 'allow profile mds'
added key for mds.a
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mds -i a -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
starting mds.a at
mkdir -p /home/mchangir/work/mchangir-ceph.git/build/dev/mds.b
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-authtool --create-keyring --gen-key --name=mds.b /home/mchangir/work/mchangir-ceph.git/build/dev/mds.b/keyring
creating /home/mchangir/work/mchangir-ceph.git/build/dev/mds.b/keyring
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring -i /home/mchangir/work/mchangir-ceph.git/build/dev/mds.b/keyring auth add mds.b mon 'allow profile mds' osd 'allow rw tag cephfs *=*' mds allow mgr 'allow profile mds'
added key for mds.b
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mds -i b -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
starting mds.b at
mkdir -p /home/mchangir/work/mchangir-ceph.git/build/dev/mds.c
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-authtool --create-keyring --gen-key --name=mds.c /home/mchangir/work/mchangir-ceph.git/build/dev/mds.c/keyring
creating /home/mchangir/work/mchangir-ceph.git/build/dev/mds.c/keyring
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring -i /home/mchangir/work/mchangir-ceph.git/build/dev/mds.c/keyring auth add mds.c mon 'allow profile mds' osd 'allow rw tag cephfs *=*' mds allow mgr 'allow profile mds'
added key for mds.c
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mds -i c -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
starting mds.c at
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring fs volume ls
[]
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring fs volume create a
Volume created successfully (no MDS daemons created)
/home/mchangir/work/mchangir-ceph.git/build/bin/ceph -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf -k /home/mchangir/work/mchangir-ceph.git/build/keyring fs authorize a client.fs_a / rwp
[client.fs_a]
key = AQC7ibJd4WUhIhAAzQS3FmIc2bIgpNEwHXqyhg==
started. stop.sh to stop. see out/* (e.g. 'tail -f out/????') for debug output.
restful urls: https://192.168.1.135:42296
w/ user/pass: admin / 743f9239-ca11-43ca-a286-c88b76c86720
----------
Here's the ceph processes listing:
$ ps aux | grep 'ceph-'
mchangir 1216 4.0 0.5 461008 88592 ? Ssl 11:04 0:03 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon -i a -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
mchangir 1320 2.6 0.5 459988 82828 ? Ssl 11:04 0:02 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon -i b -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
mchangir 1375 3.3 0.5 459984 88740 ? Ssl 11:04 0:03 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mon -i c -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
mchangir 1530 3.6 1.6 1020260 270964 ? Ssl 11:04 0:03 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mgr -i x -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
mchangir 1994 6.0 0.6 917136 111156 ? Ssl 11:04 0:04 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-osd -i 0 -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
mchangir 2545 8.5 0.7 917140 114748 ? Ssl 11:05 0:04 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-osd -i 1 -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
mchangir 3138 13.0 0.7 917148 113052 ? Ssl 11:05 0:04 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-osd -i 2 -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
mchangir 3625 0.5 0.2 403724 34416 pts/1 Sl 11:05 0:00 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mds -i a -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
mchangir 3742 0.2 0.1 337136 25476 pts/1 Sl 11:05 0:00 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mds -i b -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf
mchangir 3800 0.2 0.1 338160 26368 pts/1 Sl 11:05 0:00 /home/mchangir/work/mchangir-ceph.git/build/bin/ceph-mds -i c -c /home/mchangir/work/mchangir-ceph.git/build/ceph.conf