SvB Study

Idea

Bunch of different SvB trainings…. metric is the roc curves Xo

To do

  • [ ] FvT vs MvD vs Feynet
  • [X] Show impact of 1/3 1/10 of the dastaset
  • [X] Train with SB and SR
    • Use 3b x FvT weight for all
  • [X] Add btagging
  • [X] Add other Jet Features

Jet-less method for HH4b

Score board

variant AUC(full) AUC(SR) dSR/C0 par N(4b) N(bkg) N(ttbar) loss
DETECTOR-3b + FvT                
SR+SB 30x net 0.9580 0.9454 +0.0102 33491 1,608,312 12,787,510 476,520 0.3147
SR+SB 10x net 0.9570 0.9442 +0.0089 11051 1,608,312 12,787,510 476,520 0.3184
SR+SB 3x net 0.9528 0.9387 +0.0034 2811 1,608,312 12,787,510 476,520 0.3324
C1 SR+SB 0.9505 0.9360 +0.0007 1107 1,608,312 12,787,510 476,520 0.3397
SR-only 100x bf16 0.9454 0.9454 +0.0102 110311 1,377,354 4,861,861 217,308 0.3655
SR-only 30x net 0.9446 0.9446 +0.0094 33491 1,377,354 4,861,861 217,308 0.3680
SR-only 10x net 0.9440 0.9440 +0.0087 11051 1,377,354 4,861,861 217,308 0.3705
SR-only 3x net 0.9393 0.9393 +0.0041 2811 1,377,354 4,861,861 217,308 0.3837
C0 SR-only 0.9352 0.9352 +0.0000 1107 1,377,354 4,861,861 217,308 0.3956
C3 stat 1/3 0.9337 0.9337 -0.0015 1107 459,121 1,620,608 72,435 0.3996
C4 stat 1/10 0.9261 0.9261 -0.0092 1107 137,731 486,196 21,733 0.4321
MIXEDDATAALL + MvD                
mixed SR+SB 0.9558 0.9352 -0.0001 1107 1,608,312 4,269,684 476,520 0.2492
C2 mixed+MvD (base) 0.9294 0.9294 -0.0059 1107 1,377,354 1,480,940 217,308 0.3150
mixed 1/3 0.9226 0.9226 -0.0127 1107 459,121 493,648 72,435 0.3352
mixed 1/10 0.9116 0.9116 -0.0237 1107 137,731 148,097 21,733 0.3589
mixed rank0 0.9304 0.9304 -0.0049 1107 1,377,354 1,388,978 217,308 0.3152
mixed rank33 0.9339 0.9339 -0.0014 1107 1,377,354 1,436,139 217,308 0.3050
mixed rank77 0.9290 0.9290 -0.0062 1107 1,377,354 1,414,953 217,308 0.3177
REFERENCE                
SvB (May nominal) 0.9337 0.9337 -0.0015 1107 1,488,085 4,870,315 217,961 0.3979

2026-06-22_10-27-11_screenshot.png

Logs

20 July 2026 Monday

  • New modelling looks good!

    2026-07-20_10-27-27_screenshot.png

  • [X] Check histograms
    • Checking histograms …
    • They look good !!!
    • I think this will work.
  • [X] Update the ladder plot with the better MvD fit
    • Updating the plot with the more realistic numbers
  • Next steps…
    • [ ] calculate limits from this model.
    • [ ] Slides with plots of input variables modelling
    • [ ] How to do signal Systematics
      • ttbar tag and probe: emu + btags
        • [ ] Make skim
        • [ ] data/MC plots
        • [ ] Classifier to deal with correlations
    • [ ] Background Systematics

14 July 2026 Tuesday

  • Need to make new hists with updated MvD and the corresponding SvB
  • [>>] Update the ladder plot with the better MvD fit

13 July 2026 Monday

  • Got the Evals done… Need to make the hists

10 July 2026 Friday

  • Will use 30x all training for MvD for the background model.
  • I will then repeat the SvB Ladder studies
  • Running the eval on the 30x MvD now

09 July 2026 Thursday

  • Realizing that I need to do the overpowered MvDs before the SvB. Otherewise the perfomance gail will be overestimated… will continue running…

08 July 2026 Wednesday

  • Looks like the 1x is well modeled.
  • Need to have higher power MvD trrainings !!!

07 July 2026 Tuesday

  • Run with 1x kine SvBs
  • Reducing the histogram size for the extra hists

06 July 2026 Monday

  • Looks like the more aggressive classifier are mismodelled
  • [X] Search for over fitting.
  • Running eval on kine 1x and SR-only kine 1x
  • Made overfitting plot

2026-07-07_10-10-19_screenshot.png

01 July 2026 Wednesday

  • No effect of stacking add one or two mixed rank datasets (This was SR+SB fits)
  • Rerunning also with SR only

30 June 2026 Tuesday

  • Doing mixed statistitics test: rank00 vs (00 + 33) vs (…)
  • Crashed… debbuging .. problem with 33 (and 77) stale input files.
  • Updated them on LPC and scped to falcon
  • Rerunning

29 June 2026 Monday

  • Checking on Eval… looks like mixeddata still needs to be evaluated
  • Running eval on mixed data .. .finished
  • Checked plots on rank3 and 7… look decent
  • Plannign a scheme to add multiple rank datasets together during training.
    • Done … Now running

27 June 2026 Saturday

  • Doing the 100x Runs

2026-06-29_11-37-19_screenshot.png

26 June 2026 Friday

  • Got back the results of btagging + PNet vars

  • Launching All Jet energy vars. 

    ["pt","eta","phi","mass", "btagScore","btagPNetCvB","btagPNetCvL","btagPNetQvG", "PNetRegPtRawRes","chHEF","neHEF","chEmEF","neEmEF","muEF","nConstituents","area","rawFactor"]
  • Found a leak!!! 100% AUC Maybe rawFactor ?… yes it is !

2026-06-29_11-25-45_screenshot.png

  • Rerunning with out raw
  • Nice improvements…
  • Trying the 100x option…

24 June 2026 Wednesday

  • Running the nominal SvB traingin with the extended classifier inputs
  • Looks good…
  • Checking MvD plots… Port forwarding with ssh
  • Mixed data looks decent.. Running ranks 33 and 77
  • Starting an SvB with Btagging .. ran 2 epochs.. .looks good
  • Runing 20 epoch… results back… see improvement ….
  • Starting rank33 sample generations
  • Running at mixed w/btagging at 10x… and with

23 June 2026 Tuesday

  • MvD evaluate ran (Steering falcon jobs from lpc!)
  • [X] Make MvD histograms with new samples
  • [X] Add features to SvB
  • Now running the rest of the MvD workflow
  • Setting up the calssifier to deal with the extended features
  • Check the training the nominal vars works with the new inputs
  • Testing the extended vars
  • Turns out the extended signal inputs have not been created with the quadjetrun2 selection
  • Making them now…

22 June 2026 Monday

  • got the 1/3 results
  • Making the classifier inputs with the extended inputs
  • No need to refit MvD… No just need to reevaluate MvD!!!!

21 June 2026 Sunday

  • 100x SR+SB is back !
  • Running with 1/3 size

19 June 2026 Friday

  • Results from the 30x are in
  • Rerunning to SR 10x a couple of times with differnet seeds to measure the noise level
  • Got the noise level.. Update plots with error bars
  • Planning how to do a 100x run
  • need fp16 … implementing now

18 June 2026 Thursday

  • restarting 30x traiining
  • starting to add features to the classifier inputs … added on the dataside
  • [X] Mixed data ?
  • [X] Wire up the classifier to deal with these…
    • [X] Code it
    • [X] Test it
  • [X] Make new mixed data sets with the updated variabesl
  • Making the classifier inptus on lpc (not mixed)

17 June 2026 Wednesday

  • [X] scale up network size as dataset size increases
  • [X] Follow up on mixed data statistics relative to 3b … doing this below
  • mixed data used mixeddataallrank0
  • [X] Remake summary table with ROC in the SR
  • [X] Still looks like rank0 is smaller than 3b data…understood its b/c of the 3T+ 1M selection
  • checking sizes of mixed and 3b in
  • Running with more model parameters
  • Running with network scaled up with dataset size .. finished
  • ok I understand why the mixeddata is less than 3b.. b/c 3T+1M not 4M…
  • Now… running on rank 3 and rank 7 … finished.
  • Now running with SR only but with a biger network…
  • [>>] Update the classifier inputs
  • 10x is even better… running 30x …

16 June 2026 Tuesday

  • Checking on the jobs

      Tag Axis vs nominal Code? Status Effort
    C0 SR regions: SR → SR+SB no ready low
    C1 SRSB regions: SR → SR+SB no ready low
    C2 mixedMvD bkg: mixeddataall+MvD (4-tag/noHLG) yes (BackgroundMixed) designed medium
    C3 stat3 nominal, dataset → 1/3 (all groups) yes (–subsample) designed low–med
    C4 stat10 nominal, dataset → 1/10 (all groups) yes (–subsample) designed low–med
    C5 btag add CanJetbtagScore (on C2 base) yes + new inputs friend designed high
  • C1 done (validated wrt C0)
  • C2 running.
  • C3-C4 ran… fast!
  • Gain from C0 -> C1 2% suggests gain by adding more data
  • but no loss when reducing the stats of the training by 1/3
  • Adding the counts and the model parameters
  • Also ran the variante wrt mixeddataall
Variant AUC dAUC N(4b) N(bkg) N(ttbar) Loss
C0 SR-only 0.9352 +0.0000 1377354 4861861 217308 0.396
C1 SR+SB 0.9505 +0.0153 1608312 12787510 476520 0.340
C3 stat 1/3 0.9337 -0.0015 459121 1620608 72435 0.400
C4 stat 1/10 0.9261 -0.0092 137731 486196 21733 0.432
C2 mixed+MvD 0.9294 -0.0059 1377354 1480940 217308 0.315
mixed SR+SB 0.9548 +0.0196 1608312 4269684 476520 0.252
mixed 1/3 0.9226 -0.0127 459121 493648 72435 0.335
mixed 1/10 0.9116 -0.0237 137731 148097 21733 0.359
SvB (May nominal) 0.9337 -0.0015 1488085 4870315 217961 0.398
  • Supprising the mixed all isnt as big as the 3b…
    • [>>] Follow upp on
  • Re running witgh a SR only roc for the SR+SB runs … done

15 June 2026 Monday

  • Planning…
  • Comparision of SvB and FeynNet
  • Running in worker SvB-Studies
  • Starting to plan the various studies in claude
  • Updating the snake make for the various jobs
  • Launching the SvB with teh SR and SB
  • Debugging SR+SB jobs

Backlinks