- ceph-users - lists.ceph.io

Re: Scrubbing behind after Quincy to Reef upgrade
by Eugen Block 11 Jun '26

11 Jun '26

I feel like this has been discussed multiple times on this list, I just don't have any links at hand. I suspect mclock settings, most likely some things changed between Quincy and Reef. You could fall back to wpq instead of mclock, it's still a general recommendation at the moment, and then see if anyting improves. Zitat von Reed Dier via ceph-users <ceph-users(a)ceph.io>: > Hello all, > > TL;DR is that the number of concurrent PGs scrubbing, deep or > otherwise, has appeared to increase by about 5-10x, while the number > of PGs complaining that they haven't been scrubbed, deep or > otherwise, has continued to tick higher. >> HEALTH_WARN 217 pgs not deep-scrubbed in time; 187 pgs not scrubbed in time > > > Hoping there may be something that I must have missed in release > notes and mailing list that explains why my scrubs both exploded in > concurrency, as well as fell behind after upgrading from quincy > (17.2.9) to reef (18.2.8). > > Non-cephadm, U22.04, rather heterogenous OSD hardware. > Mix of 8T and 2T HDD, as well as 2T SSD. > HDD's have NVMe WAL/DB of various sizes depending on when they were deployed. > Mix of replicated and EC pools, as well as some replicated pools > across different device classes. > > The vast majority of the PG's that are behind on scrubbing are on EC > pools, and the vast majority of that, is our EC82 cephfs pool (40) > that holds the bulk of our stored data, and the other largest pool > is an older EC73 cephfs pool (37). > My quick and dirty approximation based on PGs last scrubbed last month. > >> ceph pg dump | grep 2026-05 | awk '{print $1" "$27}' | grep -v >> periodic | cut -d '.' -f1 | sort | uniq -c >> dumped all >> 1 17 >> 1 20 >> 116 37 >> 224 40 > > > I didn't make any changes to scrub intervals or mclock profiles > before/during/after the upgrade. >> ceph config dump | grep mclock_profile | awk '{print $4}' | uniq -c ; >> 313 balanced >> ceph config dump | grep scrub_interval >> global class:ssd advanced osd_deep_scrub_interval >> 604800.000000 >> mon advanced osd_deep_scrub_interval >> 604800.000000 >> mon.* advanced osd_deep_scrub_interval >> 604800.000000 >> mgr.* advanced osd_deep_scrub_interval >> 604800.000000 >> osd class:hdd advanced osd_deep_scrub_interval >> 604800.000000 >> osd class:ssd advanced osd_deep_scrub_interval >> 604800.000000 >> osd advanced osd_deep_scrub_interval >> 604800.000000 >> osd.* advanced osd_deep_scrub_interval >> 604800.000000 > > I've tried ceph tell osd.$osd osd_max_scrubs $more, which seems to > somewhat momentarily drive the count of > active+clean+scrubbing[+deep] PGs, but doesn't seem to make a > demonstrative difference in terms of getting ahead in the number of > PGs behind (number continues to grow). > I also looked at load15 across OSD hosts, and they don't appear to > be anywhere near the 50% threshold of osd_scrub_load_threshold > either, so I think I can rule that one out for now. > > I'm mostly curious why the change in behavior of concurrent scrubs > ballooning, and yet the number of PGs behind on scrubbing ballooning > as well, without anything actually changing. > And I'm also curious what tunables I can turn to get things back > under control for scrubbing both short and long term as I look > towards getting to squid and 24.04. > Is there an internal mechanism that triggers a deeper scrub during > first deep scrub after upgrading a major release, reef or otherwise? > > Included some graphs of scrub load over the last 60 and 365 day > period to show prior scrub load that only exceedingly rarely ever > generated a PG_NOT_[DEEP_]SCRUBBED warning, > as well as raw load average (smallest cpu count is 16, and it > doesn't even autoscale to 8, so nothing should be complaining there.) > https://imgur.com/a/rixNrCe > > Appreciate any pointers anyone can steer me towards. > Reed

1 0

Scrubbing behind after Quincy to Reef upgrade
by Reed Dier 11 Jun '26

11 Jun '26

Hello all, TL;DR is that the number of concurrent PGs scrubbing, deep or otherwise, has appeared to increase by about 5-10x, while the number of PGs complaining that they haven't been scrubbed, deep or otherwise, has continued to tick higher. > HEALTH_WARN 217 pgs not deep-scrubbed in time; 187 pgs not scrubbed in time Hoping there may be something that I must have missed in release notes and mailing list that explains why my scrubs both exploded in concurrency, as well as fell behind after upgrading from quincy (17.2.9) to reef (18.2.8). Non-cephadm, U22.04, rather heterogenous OSD hardware. Mix of 8T and 2T HDD, as well as 2T SSD. HDD's have NVMe WAL/DB of various sizes depending on when they were deployed. Mix of replicated and EC pools, as well as some replicated pools across different device classes. The vast majority of the PG's that are behind on scrubbing are on EC pools, and the vast majority of that, is our EC82 cephfs pool (40) that holds the bulk of our stored data, and the other largest pool is an older EC73 cephfs pool (37). My quick and dirty approximation based on PGs last scrubbed last month. > ceph pg dump | grep 2026-05 | awk '{print $1" "$27}' | grep -v periodic | cut -d '.' -f1 | sort | uniq -c > dumped all > 1 17 > 1 20 > 116 37 > 224 40 I didn't make any changes to scrub intervals or mclock profiles before/during/after the upgrade. > ceph config dump | grep mclock_profile | awk '{print $4}' | uniq -c ; > 313 balanced > ceph config dump | grep scrub_interval > global class:ssd advanced osd_deep_scrub_interval 604800.000000 > mon advanced osd_deep_scrub_interval 604800.000000 > mon.* advanced osd_deep_scrub_interval 604800.000000 > mgr.* advanced osd_deep_scrub_interval 604800.000000 > osd class:hdd advanced osd_deep_scrub_interval 604800.000000 > osd class:ssd advanced osd_deep_scrub_interval 604800.000000 > osd advanced osd_deep_scrub_interval 604800.000000 > osd.* advanced osd_deep_scrub_interval 604800.000000 I've tried ceph tell osd.$osd osd_max_scrubs $more, which seems to somewhat momentarily drive the count of active+clean+scrubbing[+deep] PGs, but doesn't seem to make a demonstrative difference in terms of getting ahead in the number of PGs behind (number continues to grow). I also looked at load15 across OSD hosts, and they don't appear to be anywhere near the 50% threshold of osd_scrub_load_threshold either, so I think I can rule that one out for now. I'm mostly curious why the change in behavior of concurrent scrubs ballooning, and yet the number of PGs behind on scrubbing ballooning as well, without anything actually changing. And I'm also curious what tunables I can turn to get things back under control for scrubbing both short and long term as I look towards getting to squid and 24.04. Is there an internal mechanism that triggers a deeper scrub during first deep scrub after upgrading a major release, reef or otherwise? Included some graphs of scrub load over the last 60 and 365 day period to show prior scrub load that only exceedingly rarely ever generated a PG_NOT_[DEEP_]SCRUBBED warning, as well as raw load average (smallest cpu count is 16, and it doesn't even autoscale to 8, so nothing should be complaining there.) https://imgur.com/a/rixNrCe Appreciate any pointers anyone can steer me towards. Reed

1 0

Re: tentacle 20.2.2 QE validation status
by Guillaume ABRIOUX 11 Jun '26

11 Jun '26

Hi Yuri, ceph-volume approved Regards, -- Guillaume Abrioux Software Engineer From: Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> Date: Wednesday, 10 June 2026 at 19:21 To: Nitzan Mordechai <nmordech(a)ibm.com> Cc: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] [ceph-users] Re: tentacle 20.2.2 QE validation status Still seeking approvals: fs: Venky orch/ceph-volume: Redo & Guillaume nvmeof: Aviv, Vallari On Tue, Jun 9, 2026 at 1:14 AM Nitzan Mordechai via dev <dev(a)ceph.io> wrote: > Rados approved: > https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_proje… > ------------------------------ > *From:* Yuri Weinstein via dev <dev(a)ceph.io> > *Sent:* Monday, June 8, 2026 11:54 PM > *To:* dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> > *Subject:* [EXTERNAL] Re: tentacle 20.2.2 QE validation status > > We merged several PRs listed here https: //tracker. ceph. com/issues/77140 > and re-ran several suites. Seeking approvals: rados: Radek, Nitzan rgw: > Adam Emerson fs: Venky orch/ceph-volume: Redo & Guillaume rbd: Ilya nvmeof: > Aviv, Vallari upgrade: > We merged several PRs listed here https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… > <https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> > and re-ran several suites. > Seeking approvals: > > rados: Radek, Nitzan > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redo & Guillaume > rbd: Ilya > nvmeof: Aviv, Vallari > > upgrade: Patrick, Nitzan > > On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > Details of this release are summarized here: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… > <https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> > > Release Notes - TBD > LRC upgrade - TBD > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Adam King & Guillaume > rbd: Ilya > krbd: Ilya > nvmeof: Aviv, Vallari > > upgrade: Patrick > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD > upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD > > perf-basic: PASSED > > crimson-rados: N/A > > powercycle: PASSED > > TIA > > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io Unless otherwise stated above: Compagnie IBM France Siège Social : 17, avenue de l'Europe, 92275 Bois-Colombes Cedex RCS Nanterre 552 118 465 Forme Sociale : S.A.S. Capital Social : 664 614 175,50 € SIRET : 552 118 465 03644 - Code NAF 6203Z

1 0

Re: tentacle 20.2.2 QE validation status
by Yuri Weinstein 10 Jun '26

10 Jun '26

Still seeking approvals: fs: Venky orch/ceph-volume: Redo & Guillaume nvmeof: Aviv, Vallari On Tue, Jun 9, 2026 at 1:14 AM Nitzan Mordechai via dev <dev(a)ceph.io> wrote: > Rados approved: > https://tracker.ceph.com/projects/rados/wiki/TENTACLE#v2022-additional-PR-a… > ------------------------------ > *From:* Yuri Weinstein via dev <dev(a)ceph.io> > *Sent:* Monday, June 8, 2026 11:54 PM > *To:* dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> > *Subject:* [EXTERNAL] Re: tentacle 20.2.2 QE validation status > > We merged several PRs listed here https: //tracker. ceph. com/issues/77140 > and re-ran several suites. Seeking approvals: rados: Radek, Nitzan rgw: > Adam Emerson fs: Venky orch/ceph-volume: Redo & Guillaume rbd: Ilya nvmeof: > Aviv, Vallari upgrade: > We merged several PRs listed here https://tracker.ceph.com/issues/77140 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> > and re-ran several suites. > Seeking approvals: > > rados: Radek, Nitzan > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redo & Guillaume > rbd: Ilya > nvmeof: Aviv, Vallari > > upgrade: Patrick, Nitzan > > On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/75878#note-6 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> > > Release Notes - TBD > LRC upgrade - TBD > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Adam King & Guillaume > rbd: Ilya > krbd: Ilya > nvmeof: Aviv, Vallari > > upgrade: Patrick > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD > upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD > > perf-basic: PASSED > > crimson-rados: N/A > > powercycle: PASSED > > TIA > >

1 0

June Ceph Tech Talk Recording
by Anthony Middleton 10 Jun '26

10 Jun '26

Hey Ceph Friends, If you missed yesterday's Ceph Tech Talk session on Building Ultra-Resilient Ceph Clusters with Xinnor and OSNexus, here are links to the recording and presentation slides. *Slides: https://ceph.io/assets/pdfs/tech-talks/dual_layer_durability_xinnor.pdf <https://ceph.io/assets/pdfs/tech-talks/dual_layer_durability_xinnor.pdf>* *Recording: https://youtu.be/VqKvBuemX9Q <https://youtu.be/VqKvBuemX9Q>* Our next session is in the works for July 8th on the topic of GPU-initiated IO to Ceph. I'll share more details as things progress. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: CephFS large amounts of dark data
by Anthony D'Atri 09 Jun '26

09 Jun '26

> ). >>> >>> There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used consistent with 8+3 EC. >>> The layout for most files: >>> ceph.dir.layout="stripe_unit=16777216 stripe_count=1 object_size=16777216 pool=cephfs_data_echdd" >>> >>> But there is 1.4PiB discrepancy between the pool and the filesystem Do you have scrubs enabled? Which if any non-default config options do you have set? Any undersized or degraded or backfilling PGs? Which Ceph release? Do you have a sizable fraction of small files? If you’re running Squid or earlier or don’t have EC optimizations enabled, even a tiny file will allocate a multiple of 11*16=176 KB. An 129KB file will consume 352KB, etc. If I understand those layout options correctly. If that’s what’s going on, going to Tentacle with EC optimizations would gain you some efficiency for files newly [re]written. You could also migrate small files to a replicated pool. >>> which I cannot explain and I suspect there are a lot of orphan objects there. I have run mds scrub on / and ~mdsdir as well. There is some mds damage on some old small files (~400 files), which I do not think it's relevant here. >>> >> Hello, >> We had a similar issue last year with a group of users that created and removed files at a very high rate. >> >> Have you read https://docs.clyso.com/docs/kb/cephfs/#cephfs-pool-data-usage-growth-withou… ? >> >> We increased the purge rate parameters (very) aggressively to get back to a comfortable situation (i.e. not a pool w/ near full warnings). >> >> >> Loïc. > > > -- > _____________________________________________________________ > prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si > Department of Experimental High Energy Physics - F9 > Jozef Stefan Institute, Jamova 39, P.o.Box 3000 > SI-1001 Ljubljana, Slovenia > Tel.: +386-1-477-3674 Fax: +386-1-477-3166 > ------------------------------------------------------------- > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: CephFS large amounts of dark data
by Gregory Farnum 09 Jun '26

09 Jun '26

Do you have any snapshots? There's also generally some amount of inflation with EC and small files, but I don't think it should reach (7/5.6=)25%, and since you're using 16MB objects I presume most of your files are large. -Greg On Tue, Jun 9, 2026 at 7:02 AM Andrej Filipčič via ceph-users <ceph-users(a)ceph.io> wrote: > > > thanks for this info, but our purge queue is more or less empty > "purge_queue": { > "pq_executing_ops": 2, > "pq_executing_ops_high_water": 13659, > "pq_executing": 1, > "pq_executing_high_water": 500, > "pq_executed_ops": 127095, > "pq_executed": 50471, > "pq_item_in_journal": 2 > }, > > anyway, I raised the parameters to see if it helps. > > Best, > Andrej > > > On 9. 6. 2026 14:57, Loïc Tortay via ceph-users wrote: > > On 09/06/2026 14:10, Andrej Filipcic via ceph-users wrote: > >> > >> Hi, > >> > >> I have a problem with large amounts of dark data/orphans on cephfs > >> data pool. Data is stored mostly on a 8+3 EC pool (cephfs_data_echdd). > >> > >> There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M > >> files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used > >> consistent with 8+3 EC. > >> The layout for most files: > >> ceph.dir.layout="stripe_unit=16777216 stripe_count=1 > >> object_size=16777216 pool=cephfs_data_echdd" > >> > >> But there is 1.4PiB discrepancy between the pool and the filesystem > >> which I cannot explain and I suspect there are a lot of orphan > >> objects there. I have run mds scrub on / and ~mdsdir as well. There > >> is some mds damage on some old small files (~400 files), which I do > >> not think it's relevant here. > >> > > Hello, > > We had a similar issue last year with a group of users that created > > and removed files at a very high rate. > > > > Have you read > > https://docs.clyso.com/docs/kb/cephfs/#cephfs-pool-data-usage-growth-withou… > > ? > > > > We increased the purge rate parameters (very) aggressively to get back > > to a comfortable situation (i.e. not a pool w/ near full warnings). > > > > > > Loïc. > > > -- > _____________________________________________________________ > prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si > Department of Experimental High Energy Physics - F9 > Jozef Stefan Institute, Jamova 39, P.o.Box 3000 > SI-1001 Ljubljana, Slovenia > Tel.: +386-1-477-3674 Fax: +386-1-477-3166 > ------------------------------------------------------------- > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: CephFS large amounts of dark data
by Andrej Filipčič 09 Jun '26

09 Jun '26

thanks for this info, but our purge queue is more or less empty "purge_queue": { "pq_executing_ops": 2, "pq_executing_ops_high_water": 13659, "pq_executing": 1, "pq_executing_high_water": 500, "pq_executed_ops": 127095, "pq_executed": 50471, "pq_item_in_journal": 2 }, anyway, I raised the parameters to see if it helps. Best, Andrej On 9. 6. 2026 14:57, Loïc Tortay via ceph-users wrote: > On 09/06/2026 14:10, Andrej Filipcic via ceph-users wrote: >> >> Hi, >> >> I have a problem with large amounts of dark data/orphans on cephfs >> data pool. Data is stored mostly on a 8+3 EC pool (cephfs_data_echdd). >> >> There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M >> files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used >> consistent with 8+3 EC. >> The layout for most files: >> ceph.dir.layout="stripe_unit=16777216 stripe_count=1 >> object_size=16777216 pool=cephfs_data_echdd" >> >> But there is 1.4PiB discrepancy between the pool and the filesystem >> which I cannot explain and I suspect there are a lot of orphan >> objects there. I have run mds scrub on / and ~mdsdir as well. There >> is some mds damage on some old small files (~400 files), which I do >> not think it's relevant here. >> > Hello, > We had a similar issue last year with a group of users that created > and removed files at a very high rate. > > Have you read > https://docs.clyso.com/docs/kb/cephfs/#cephfs-pool-data-usage-growth-withou… > ? > > We increased the purge rate parameters (very) aggressively to get back > to a comfortable situation (i.e. not a pool w/ near full warnings). > > > Loïc. -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------

1 0

Re: CephFS large amounts of dark data
by Loïc Tortay 09 Jun '26

09 Jun '26

On 09/06/2026 14:10, Andrej Filipcic via ceph-users wrote: > > Hi, > > I have a problem with large amounts of dark data/orphans on cephfs data > pool. Data is stored mostly on a 8+3 EC pool (cephfs_data_echdd). > > There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M > files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used > consistent with 8+3 EC. > The layout for most files: > ceph.dir.layout="stripe_unit=16777216 stripe_count=1 > object_size=16777216 pool=cephfs_data_echdd" > > But there is 1.4PiB discrepancy between the pool and the filesystem > which I cannot explain and I suspect there are a lot of orphan objects > there. I have run mds scrub on / and ~mdsdir as well. There is some mds > damage on some old small files (~400 files), which I do not think it's > relevant here. > Hello, We had a similar issue last year with a group of users that created and removed files at a very high rate. Have you read https://docs.clyso.com/docs/kb/cephfs/#cephfs-pool-data-usage-growth-withou… ? We increased the purge rate parameters (very) aggressively to get back to a comfortable situation (i.e. not a pool w/ near full warnings). Loïc. -- | Loīc Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

1 0

CephFS large amounts of dark data
by Andrej Filipcic 09 Jun '26

09 Jun '26

Hi, I have a problem with large amounts of dark data/orphans on cephfs data pool. Data is stored mostly on a 8+3 EC pool (cephfs_data_echdd). There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used consistent with 8+3 EC. The layout for most files: ceph.dir.layout="stripe_unit=16777216 stripe_count=1 object_size=16777216 pool=cephfs_data_echdd" But there is 1.4PiB discrepancy between the pool and the filesystem which I cannot explain and I suspect there are a lot of orphan objects there. I have run mds scrub on / and ~mdsdir as well. There is some mds damage on some old small files (~400 files), which I do not think it's relevant here. ceph was created with 17.2.6 and runs 19.2.3. Do you might have any suggestions how to fix this? Thanks, Andrej spod01 ~# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 11 PiB 1.5 PiB 9.7 PiB 9.7 PiB 86.28 nvme 363 TiB 356 TiB 6.8 TiB 6.8 TiB 1.87 ssd 3.5 TiB 3.0 TiB 478 GiB 478 GiB 13.37 TOTAL 12 PiB 1.9 PiB 9.7 PiB 9.7 PiB 83.68 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 850 MiB 213 2.5 GiB 0 178 TiB cephfs_data 2 256 28 GiB 134.71M 86 GiB 2.86 972 GiB cephfs_metadata 3 64 66 GiB 6.66M 199 GiB 6.38 972 GiB cephfs_data_echdd 19 4096 7.0 PiB 594.70M 9.7 PiB 94.90 388 TiB cephfs_data_ecnvme 26 64 1.2 TiB 2.84M 1.7 TiB 0.48 251 TiB ... -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------

1 0