Run3 FvT

• Re-Running histogrmas on falcon

./run_container snakemake --snakefile coffea4bees/workflows/Snakefile_Run3.smk --profile software/snakemake/profiles/falcon --config mode=quadjet_run2 --dry-run --jobs 20 --rerun-triggers mtime --forcerun make_histograms make_histograms_wJCM make_histograms_FvT -- all_histograms all_FvT_histograms

• The big SnakeMake job finished !
• FvT looks plausible:

• Results look OK… FvT weights still cant fit FeynNet output

• Will Try to Run all on falcon
• Runnign the SvB for HH4b signal… lpc… cmslpc323

./run_container snakemake --profile software/snakemake/profiles/lpc --snakefile coffea4bees/workflows/Snakefile_SvB_friendtrees_Run3.smk --cores 4 --logger snkmt --retries 2 --rerun-triggers mtime --forcerun make_SvB_friendtrees_HH

• Running…faild b/c ran out of memory … rerunning… Finished
• Need to remake the classifier inputs … on falcon ..

• Breaking out a separate Snakemake workflow for the classifier inputs

  ./run_container snakemake --snakefile coffea4bees/workflows/Snakefile_classifier_inputs_Run3.smk --profile software/snakemake/profiles/falcon --config mode=quadjet_run2 run_on_condor=False --forceall --jobs 20 --logger snkmt
  

• Errors slurm timing configs and output logfile paths … fixing… rerunning
• getting the condor and slurm errors cleared
• Doing full Run3

./run_container snakemake --snakefile coffea4bees/workflows/Snakefile_Run3.smk --profile software/snakemake/profiles/falcon --config mode=quadjet_run2 classifier=FvT --jobs 20 --logger snkmt --  all_with_training

• [>>] Need a way to submit many sub jobs in slurm on falcon … Check with AE

• Run with new SvB_FeynNet friends

• Running on tmux on 343

  ./run_container snakemake --profile software/snakemake/profiles/lpc --snakefile coffea4bees/workflows/Snakefile_Run3.smk --config mode=quadjet_run2 --cores 4  --logger snkmt  --rerun-triggers mtime -- all_FvT_histograms
  

• One job failed b/c Address already in use…fixed…
• Compare new and old… not much different:

• Now testing with the 3T1M selection …Just changes the norm by ~factor of w
• Fixing FeynNet … should be using P(Sig vs Bkg) = P(Sig) / (P(Sig) + P(Bkg))

• Looks like I will have to rederive the FriendTrees to fix this

  ./run_container snakemake --profile software/snakemake/profiles/lpc --snakefile coffea4bees/workflows/Snakefile_SvBFeynNet_friendtrees_Run3.smk --cores 4  --forceall  --logger snkmt
  

• Crash from another address collision … will have to rerun … running on cmslpc335… finished
• Re-running the hists… taking a while

• Rerunning failed jobs on cmslpc373 tmux
• Looks good

• FvT NaN chase down

• Found it. The nominal HCR input friend tree has all-zero CanJet kinematics and ancillary for these events:

  CanJet_pt=[0.0, 0.0, 0.0, 0.0] ancillary=[xW=0, nSelJets=0, xbW=0, year=23]

  Compare to the quadjet_run2 HCR_input where the same events had valid [154, 66, 55, 31]. So in the nominal classifier_{inputsRun3.json}, these events were written as zeros — they failed the nominal HCR-input writer's selection but pass the quadjet_run2 one.

  The FvT model NaNs because it gets fed all-zero inputs (log(0)-clipped to log(1e-6) plus a ghost-batch-norm divide somewhere → NaN). The standalone forward I ran with the quadjet_run2 HCR_input gave valid output because it had real kinematics.

  Let me confirm with a direct dump of both HCR_input chunks side-by-side:

• Thinking I may have used the wrong classifer inputs file…should have used the MvD one…yes used inconsistent file in train and evaluate!
• Will re-evaluate.

• Running in tmux on falcon

  ./run_container snakemake --snakefile coffea4bees/workflows/Snakefile_Run3.smk output/Run3_quadjet_run2/FvT/evaluate.log --config mode=quadjet_run2 --logger snkmt --cores 1 --dry-run
  

• Evaluate finished
• Checked that the NaNS are gone!
• Rerunning the histograms… in tmux cmslpc373

• Tryihng Evaluation on falcon

• running dryrun

  ./run_container snakemake --snakefile coffea4bees/workflows/Snakefile_Run3.smk output/Run3_quadjet_run2/FvT/evaluate.done --config mode=quadjet_run2 --logger snkmt --cores 1
  

• have to log into to rogues to install pixi for the first time….
• Then have to submit via falcon
• Failures… barista error… now voms expired ….
• Running in tmux….RUNNING

pixi run snkmt console

• works on barista … worth setting up fancier monitor …. probably not yet
• Another crash… something with Name and ellipses failingin the merge step…. rerunning
• Finished… adding FvT hists to snakemake
• Runnign on … cmslpc326… finished
• Seeing nans in FvT output !!!
• Got some events example… need to debug .. (w/cluade)
  • [X] Are there nan classifier inputs ?… No
  • [X] Check the traingin files

Definitive answer: the inputs are clean. All HCR input features for both offender events are finite and plausibly-normal — no NaN, no Inf. So the NaN is arising inside the FvT forward pass on valid inputs.

How can xW be negative?… easy

• For now set Nans to 1.0
• Rerunning on cmslpc313… Ran
• Looks like SvB_FenyNet does nt close from 3b ! … does for SvB_MA though …
• Solved in future... mis match in classifier inputs file used for training and evaluation

• Writting Run3 Snakemake file

• Running on tmux cmslpc372

  ./run_container snakemake --snakefile coffea4bees/workflows/Snakefile_Run3.smk --config mode=quadjet_run2 --logger snkmt --cores 4 --dry-run --printsh
  

• Running … jcm seems off

./run_container python coffea4bees/analysis/jcm_tools/make_jcm_weights.py  -o test/jcm_for_Run3   -i output/Run3_quadjet_run2/histAll_Run3_quadjet_run2.coffea       -r SB         --jcm_config coffea4bees/analysis/jcm_tools/metadata/nominal_jcm_config.yml 

• Fixed… just a problem with the tagJets plot… fixed
• But now seems like the JCM prediction does not agree with the weighted 3b events … No It was b/c I was not subtracting the threeTag ttbar (need to use plotsAllNoFvT_noSignal.yml
• Should also run FvT
• Is the TTbar different between 3b vs 4b and mixed no MvD vs 4b ? … Not its an effect of the normalization
• Have a good first pass of the MvD slides