- ceph-users - lists.ceph.io

Samsung PM883 3.84TB SSD performance
by mj 19 Jan '21

19 Jan '21

Hi, We purchased Samsung PM883 3.84TB SSDs to be used as BlueStore SSDs in our cluster. I ran some benchmarks (write cache disabled, and with -sync=1 and -fsync=1) but with my little understandig, the results look terribly bad. (iops around 82!) Reading pages like yourcmc.ru, we should at least aim for > 1000s IOPS. (like 10000 up to 20000) Here is our complete output: > root@pve:~# hdparm -W 0 /dev/sdd > > /dev/sdd: > setting drive write-caching to 0 (off) > write-caching = 0 (off) > root@pve:~# fio -ioengine=libaio -fsync=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdd > test: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=32 > fio-2.16 > Starting 1 process > Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=3268778: Tue Jan 19 14:13:44 2021 > write: io=19728MB, bw=336674KB/s, iops=82, runt= 60003msec > slat (usec): min=1853, max=4652, avg=2428.94, stdev=361.92 > clat (msec): min=9, max=652, avg=376.87, stdev=22.88 > lat (msec): min=11, max=655, avg=379.30, stdev=22.94 > clat percentiles (msec): > | 1.00th=[ 363], 5.00th=[ 367], 10.00th=[ 371], 20.00th=[ 371], > | 30.00th=[ 371], 40.00th=[ 375], 50.00th=[ 375], 60.00th=[ 379], > | 70.00th=[ 379], 80.00th=[ 383], 90.00th=[ 388], 95.00th=[ 392], > | 99.00th=[ 404], 99.50th=[ 420], 99.90th=[ 611], 99.95th=[ 635], > | 99.99th=[ 652] > lat (msec) : 10=0.02%, 50=0.06%, 100=0.10%, 250=0.24%, 500=99.25% > lat (msec) : 750=0.32% > cpu : usr=2.95%, sys=28.30%, ctx=47835, majf=0, minf=26 > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%, >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% > issued : total=r=0/w=4932/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 > latency : target=0, window=0, percentile=100.00%, depth=32 > > Run status group 0 (all jobs): > WRITE: io=19728MB, aggrb=336674KB/s, minb=336674KB/s, maxb=336674KB/s, mint=60003msec, maxt=60003msec > > Disk stats (read/write): > sdd: ios=41/39330, merge=0/4995672, ticks=8/231816, in_queue=231440, util=79.43% > root@pve:~# fio -ioengine=libaio -sync=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdd > test: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=32 > fio-2.16 > Starting 1 process > Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=3270282: Tue Jan 19 14:14:58 2021 > write: io=19468MB, bw=332248KB/s, iops=81, runt= 60001msec > slat (usec): min=11658, max=14287, avg=12318.59, stdev=371.39 > clat (usec): min=8, max=400078, avg=380931.63, stdev=19107.88 > lat (msec): min=11, max=412, avg=393.25, stdev=19.16 > clat percentiles (msec): > | 1.00th=[ 371], 5.00th=[ 375], 10.00th=[ 375], 20.00th=[ 375], > | 30.00th=[ 375], 40.00th=[ 379], 50.00th=[ 379], 60.00th=[ 383], > | 70.00th=[ 388], 80.00th=[ 392], 90.00th=[ 392], 95.00th=[ 396], > | 99.00th=[ 396], 99.50th=[ 400], 99.90th=[ 400], 99.95th=[ 400], > | 99.99th=[ 400] > lat (usec) : 10=0.02% > lat (msec) : 20=0.02%, 50=0.04%, 100=0.08%, 250=0.25%, 500=99.59% > cpu : usr=2.22%, sys=30.01%, ctx=53455, majf=0, minf=25 > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%, >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% > issued : total=r=0/w=4867/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 > latency : target=0, window=0, percentile=100.00%, depth=32 > > Run status group 0 (all jobs): > WRITE: io=19468MB, aggrb=332248KB/s, minb=332248KB/s, maxb=332248KB/s, mint=60001msec, maxt=60001msec > > Disk stats (read/write): > sdd: ios=41/38830, merge=0/4931664, ticks=4/203944, in_queue=203700, util=79.55% > > The disk seems to be properly detected properly by the OS: > [4145961.899584] scsi 0:0:16:0: Direct-Access ATA SAMSUNG MZ7LH3T8 904Q PQ: 0 ANSI: 6 > [4145961.899596] scsi 0:0:16:0: SATA: handle(0x001a), sas_addr(0x500304801ef80883), phy(3), device_name(0x0000000000000000) > [4145961.899598] scsi 0:0:16:0: enclosure logical id (0x500304801ef808bf), slot(3) > [4145961.899599] scsi 0:0:16:0: enclosure level(0x0000), connector name( ) > [4145961.899649] scsi 0:0:16:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y) > [4145961.900471] sd 0:0:16:0: Power-on or device reset occurred > [4145961.900544] sd 0:0:16:0: Attached scsi generic sg3 type 0 > [4145961.902367] sd 0:0:16:0: [sdd] 7501476528 512-byte logical blocks: (3.84 TB/3.49 TiB) > [4145961.902369] sd 0:0:16:0: [sdd] 4096-byte physical blocks > [4145961.904264] sd 0:0:16:0: [sdd] Write Protect is off > [4145961.904266] sd 0:0:16:0: [sdd] Mode Sense: 9b 00 10 08 > [4145961.904756] sd 0:0:16:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA > [4145961.915996] sd 0:0:16:0: [sdd] Attached SCSI disk Anyone with a idea what we could be doing wrong? Or are these disks really unsuitable for OSD use? MJ

2 1

Dashboard : Block image listing and infos
by Gilles Mocellin 18 Jan '21

18 Jan '21

Hello Cephers, On a new cluster, I only have 2 RBD block images, and the Dashboard doesn't manage to list them correctly. I have this message : Warning Displaying previously cached data for pool veeam-repos. Sometime it disappears, but as soon as I reload or return to the listing page, it's there. What I've seen, is a high CPU load due to ceph-mgr on the active manager. And also stack-traces like this : 2021-01-15T14:41:12.061+0100 7f7f3fec4700 0 [dashboard ERROR exception] Dashboard Exception Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 94, in dashboard_exception_handler return handler(*args, **kwargs) File "/usr/lib/python3/dist-packages/cherrypy/_cpdispatch.py", line 60, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 666, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 861, in wrapper return func(*vpath, **params) File "/usr/lib/python3.6/contextlib.py", line 52, in inner return func(*args, **kwds) File "/usr/lib/python3.6/contextlib.py", line 52, in inner return func(*args, **kwds) File "/usr/share/ceph/mgr/dashboard/controllers/rbd.py", line 86, in list return self._rbd_list(pool_name) File "/usr/share/ceph/mgr/dashboard/controllers/rbd.py", line 76, in _rbd_list status, value = RbdService.rbd_pool_list(pool) File "/usr/share/ceph/mgr/dashboard/tools.py", line 254, in wrapper return rvc.run(fn, args, kwargs) File "/usr/share/ceph/mgr/dashboard/tools.py", line 242, in run raise ViewCacheNoDataException() dashboard.exceptions.ViewCacheNoDataException: ViewCache: unable to retrieve data Also that, since I changed some features back and forth on one image : 2021-01-18T11:13:26.383+0100 7f00199ca700 0 [dashboard ERROR frontend.error] (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/#/block/rbd/edit/veeam- repos%252Fveeam-repo2-vol1): Cannot read property 'features_name' of undefined TypeError: Cannot read property 'features_name' of undefined at https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/1.9e79c41bbaed982a50af.js:1:121… at Array.forEach (<anonymous>) at R.deepBoxCheck (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/1.9e79c41bbaed982a50af.js:1:120…) at R.featureFormUpdate (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/1.9e79c41bbaed982a50af.js:1:121…) at https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/1.9e79c41bbaed982a50af.js:1:119… at d.a [as _next] (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/main.c43d13b597196a5f022f.js:2:…) at d.__tryOrUnsub (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/main.c43d13b597196a5f022f.js:2:…) at d.next (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/main.c43d13b597196a5f022f.js:2:…) at l._next (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/main.c43d13b597196a5f022f.js:2:…) at l.next (https://fidcl-mrs4-sto-sds.fidcl.cloud:8443/main.c43d13b597196a5f022f.js:2:…) But that's perhaps just becaus I open an Edit window on the image and it does not have the datas. The Edit window is empty, and I can't edit things, especially, I wan't to resize the image. Finally, I find a similar bug already registered here, but that seems resolved for the owner : https://tracker.ceph.com/issues/45308 Mmmm, as I read carrefully, it'about cephfs not rbd in that stacktrace... Am I the only one ? Is there a work around ? I really need the dashboard to be usable, because I want to delegate a maximum of operations to people that don't need to have the rights to connect to machines to use CLI. And also don't have the skills to do so. Have a good day, -- Gilles

1 0

RBD image size in prometheus
by Seena Fallah 18 Jan '21

18 Jan '21

Hi, Is there any reason why RBD image size isn't exported in the Prometheus module? Thanks.

1 0

PG inconsistent with empty inconsistent objects
by Seena Fallah 17 Jan '21

17 Jan '21

Hi, I'm facing something strange! One of the PGs in my pool got inconsistent and when I run `rados list-inconsistent-obj $PG_ID --format=json-pretty` the `inconsistents` key was empty! What is this? Is it a bug in Ceph or..? Thanks.

3 4

Ceph 15.2.3 on Ubuntu 20.04 with odroid xu4 / python thread Problem
by Dominik H 16 Jan '21

16 Jan '21

Hi all, i try to run ceph client tools on an odroid xu4 (armhf) with Ubuntu 20.04 on python 3.8.5. Unfortunately there is the following error on each "ceph" command (even in ceph --help) Traceback (most recent call last): File "/usr/bin/ceph", line 1275, in <module> retval = main() File "/usr/bin/ceph", line 981, in main cluster_handle = run_in_thread(rados.Rados, File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 1342, in run_in_thread raise Exception("timed out") Exception: timed out With this Server I access an existing Ceph-Cluster with the same hardware. I checked the code part, there is just a thread start and a join (waiting for finish a RadosThread). Maybe this is a python error in combination with armhf architecture? Maybe someone can help. Thanks and greetings Dominik

3 8

librbd::DiffIterate: diff_object_map: failed to load object map rbd_object_map.
by Rafael Diaz Maurin 15 Jan '21

15 Jan '21

Hello cephers, I run Nautilus (14.2.15) Here is my context : each night a script take a snapshot from each RBD volume in a pool (all the disks of the VMs hosted) on my ceph production cluster. Then each snapshot is exported (rbd export-diff | rbd import-diff in SSH) towards my ceph bakup cluster. 0. Yesterday, and for the firt time, I had this error when I ran ''rbd -p MY_POOL du'' on the ceph backup cluster : 021-01-14 11:41:29.294 7f1f2fbcf0c0 -1 librbd::DiffIterate: diff_object_map: failed to load object map *rbd_object_map.781776950f8dd5.000000000000b740* 1. I tried to find the image concerned by the error, and I find this one, with a slight diffence with the number of the object-map : rbd image 'MY_IMAGE': size 30 GiB in 7680 objects order 22 (4 MiB objects) snapshot_count: 32 id: 781776950f8dd5 block_name_prefix: *rbd_data.781776950f8dd5* format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, operations op_features: snap-trash flags: create_timestamp: Thu Mar 19 01:00:27 2020 access_timestamp: Thu Mar 19 01:00:27 2020 modify_timestamp: Thu Jan 14 01:42:23 2021 2. Then I ran a check wich not show any error rbd -p MY_POOL object-map check "MY_IMAGE" Object Map Check: 100% complete...done. 3. Then, I still ran a rebuild rbd -p MY_POOL object-map rebuild "MY_IMAGE" Object Map Rebuild: 100% complete...done. 4. But the error is still here : 2021-01-15 09:33:58.775 7fa088e350c0 -1 librbd::DiffIterate: diff_object_map: failed to load object map rbd_object_map.781776950f8dd5.000000000000b740 Do someone know how to find the the image concerned by the object-map error and fix this error. Thank you for your help, Rafael

3 5

Python API mon_comand()
by Robert Sander 15 Jan '21

15 Jan '21

Hi, I am trying to get some statistics via the Python API but fail to run the equivalent of "ceph df detail". One the command line I get: # ceph -f json df |jq .pools[0] { "name": "rbd", "id": 1, "stats": { "stored": 27410520278, "objects": 6781, "kb_used": 80382849, "bytes_used": 82312036566, "percent_used": 0.1416085809469223, "max_avail": 166317473792 } } # ceph -f json df detail |jq .pools[0] { "name": "rbd", "id": 1, "stats": { "stored": 27410520278, "objects": 6781, "kb_used": 80382849, "bytes_used": 82312036566, "percent_used": 0.1416085809469223, "max_avail": 166317473792, "quota_objects": 0, "quota_bytes": 0, "dirty": 6781, "rd": 309130743, "rd_bytes": 327278814208, "wr": 155492443, "wr_bytes": 155528225792, "compress_bytes_used": 0, "compress_under_bytes": 0, "stored_raw": 82231558358, "avail_raw": 498952444191 } } In Python I just get an error: # python Python 2.7.16 (default, Oct 10 2019, 22:02:15) [GCC 8.3.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import json >>> import rados >>> cluster = rados.Rados(conffile='/etc/ceph/ceph.conf') >>> cluster.connect() >>> cluster.mon_command(json.dumps({'prefix': 'df', 'format': 'json'}), b'') (0, '{"stats":{"total_bytes":643914731520,"total_avail_bytes":550794231808,"total_used_bytes":86678048768,"total_used_raw_bytes":93120499712,"total_used_raw_ratio":0.14461618661880493,"num_osds":6,"num_per_pool_osds":6},"stats_by_class":{"hdd":{"total_bytes":643914731520,"total_avail_bytes":550794231808,"total_used_bytes":86678048768,"total_used_raw_bytes":93120499712,"total_used_raw_ratio":0.14461618661880493}},"pools":[{"name":"rbd","id":1,"stats":{"stored":27410520278,"objects":6781,"kb_used":80382849,"bytes_used":82312036566,"percent_used":0.1416085809469223,"max_avail":166317473792}},{"name":"cephfs_data","id":3,"stats":{"stored":1282414464,"objects":307,"kb_used":3757248,"bytes_used":3847421952,"percent_used":0.0076519949361681938,"max_avail":166317473792}},{"name":"cephfs_metadata","id":4,"stats":{"stored":458803,"objects":22,"kb_used":2693,"bytes_used":2757248,"percent_used":5.5260434237425216e-06,"max_avail":166317473792}}]}\n', u'') >>> cluster.mon_command(json.dumps({'prefix': 'df detail', 'format': 'json'}), b'') (-22, '', u'command not known') >>> Anything I can do to get the output of "ceph df detail" via Python API? I would like to have the stats fields "rd", "wr", "rd_bytes" and "wr_bytes" per pool. Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 1

.rgw.root was created wit a lot of PG
by Szabo, Istvan (Agoda) 15 Jan '21

15 Jan '21

Hi, Originally this pool was created with 512PG which makes couple of OSDs having 500PG 😲 What is the safe steps to copy over this pool? These are the files in this pool: default.realm period_config.f320e60d-8cff-4824-878e-c316423cc519 periods.18d63a25-8a50-4e17-9561-d452621f62fa.latest_epoch default.zonegroup.f320e60d-8cff-4824-878e-c316423cc519 zone_info.ba16656f-2191-40bb-bc39-9f19448d215d periods.6605bb4c-2226-4509-a3be-d5c95300fe14.1 default.zone.f320e60d-8cff-4824-878e-c316423cc519 zonegroup_info.f47a81ba-b214-4b8d-9b0e-84c14bc153cf periods.f320e60d-8cff-4824-878e-c316423cc519:staging realms.f320e60d-8cff-4824-878e-c316423cc519.control periods.6605bb4c-2226-4509-a3be-d5c95300fe14.latest_epoch realms.f320e60d-8cff-4824-878e-c316423cc519 realms_names.default periods.18d63a25-8a50-4e17-9561-d452621f62fa.1 zone_names.default zonegroups_names.default It’s single site cluster. Do I need to stop rados gateway? After I would follow the following steps: ceph osd pool create .rgw.root.new 8 rados cppool .rgw.root .rgw.root.new ceph osd pool delete .rgw.root .rgw.root --yes-i-really-really-mean-it ceph osd pool rename .rgw.root.new .rgw.root ceph osd pool application enable .rgw.root rgw Just not sure do I need to create outage request or not due to the rados stop. Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3

Centos 8 2021 with ceph, how to move forward?
by Szabo, Istvan (Agoda) 14 Jan '21

14 Jan '21

Hi, Just curious how you guys move forward with this Centos 8 change. We just finished installing our full multisite cluster and looks like we need to change the operating system. So curious if you are using centos 8 with ceph, where you are going to move forward. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

6 6

radosgw-admin sync status takes ages to print output
by Szabo, Istvan (Agoda) 14 Jan '21

14 Jan '21

Hello, I have a 3 DC octopus Multisite setup with bucket sync policy applied. I have 2 buckets where I’ve set the shard 24.000 and the other is 9.000 because they want to use 1 bucket but with a huge amount of objects (2.400.000.000 and 900.000.000) and in case of multisite we need to preshard the buckets as it is in the documentation. Do I need to fine tune something on the syncing to make this query faster? This is the output after 5-10 minutes query time not sure is it healthy or good or not to be honest, haven’t really find any good explanation about the output in the ceph documentation. From the master zone I can’r reallt even query because timed out, but in secondary zone can see this: radosgw-admin sync status realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm) zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data) zone 61c9d940-fde4-4bed-9389-edc8d7741817 (sin) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-14T12:01:00.131104+0700 [11] source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-14T12:05:26.879014+0700 [98] Hope I can find some expert in the multisite area 😊 Thank you in advance. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3