4b Background from Mixed-data

• Making rank 7 histograms
• Starting the rank 3 MvD on falcon… running …
• Making rank 3 MvD Histograms
• Starting the rank 7 MvD on falcon… running …

Bam!!!! Rank0 ~ Rank3 ~ Rank7… can make 50 datasets this good.

• Rank 0 MvD training finished..
• ROCs look good
• Evaluate fails… fix in wrong input friend tree in config… running again on cmslpcgpu1… finished

• Running plots … on cmslpc370

  ./run_container snakemake --profile software/snakemake/profiles/lpc --snakefile coffea4bees/workflows/Snakefile_Run3MvD.smk output/Run3_MvD_quadjet_run2_rank0/plots_MvD/plots_done.txt --cores 4 --config dataset_name=mixeddata_all_rank0 mode=quadjet_run2 --rerun-triggers mtime --logger snkmt
  

• Getting memory pressure… resubmitting
• Rerunning some crashed jobs
• [ ] What is the input JCM used for ?
• Should redo JCM and FvT-4M for mixing
• Fits look reasonable…
• [ ] Could the pseudoTag assigment be biasing the mixing?
  • Never mixing to event with more than 4 bjets
• Making rank 3 mixed datasets on cmslpc370

./run_container snakemake --profile software/snakemake/profiles/lpc --snakefile coffea4bees/workflows/Snakefile_Run3_make_mixeddata.smk --cores 4 --config 'default_rank=[3,3]' mode=quadjet_run2

• Updating slides

• Make rule graph by adding --rulegraph put the output text in text.txt then…

  cat test.txt |dot -Tpng > rulegraph.png
  

• Debugging rank 3 running… finished plots

  ./run_container snakemake --profile software/snakemake/profiles/lpc --snakefile coffea4bees/workflows/Snakefile_Run3MvD.smk --cores 4 --config dataset_name=mixeddata_all_rank3_3 mode=quadjet_run2
  

• Rank 3 looks damn close to rank0

  

• [ ] Where can I turn off ttbar making
• Starting rank 7 on cmslpc371

./run_container snakemake --profile software/snakemake/profiles/lpc --snakefile coffea4bees/workflows/Snakefile_Run3_make_mixeddata.smk --cores 4 --config 'default_rank=[7,7]' mode=quadjet_run2 --rerun-triggers mtime --logger snkmt

• starting MvD on new mixed sample rank0
• Running all on falcon… no I will not…

• Running all up to training with:… killed b/c SvB friends … see below

  ./run_container snakemake --snakefile coffea4bees/workflows/Snakefile_Run3MvD.smk --cores 4 --config dataset_name=mixeddata_all_rank0 mode=quadjet_run2 --dry-run
  

• Runjning on cmslpc306…
• Realized I also need to make SvB friend trees.
• Wiring that in…

• Running SvB friends on cmslpc340 … finished.

  ./run_container snakemake --snakefile coffea4bees/workflows/Snakefile_Run3_make_mixeddata.smk --cores 4 --rerun-triggers mtime --config mode=quadjet_run2 --logger snkmt
  

• Back to MvD… rerunning… on cmslpc306.. finished
• [X] Split out the histogrmas that have to be remade from those that done
• Running training

./run_container snakemake --snakefile coffea4bees/workflows/Snakefile_Run3MvD.smk output/Run3_MvD_quadjet_run2_rank0/MvD/evaluate.done output/Run3_MvD_quadjet_run2_rank0/MvD/analyze.done --jobs 4 --config  dataset_name=mixeddata_all_rank0 mode=quadjet_run2   --logger snkmt

• Jobs queued

  [falcon] /home/export/jalison/work/barista > squeue 
       JOBID PARTITION         NAME       USER ST       TIME  NODES TRES_PER_NODE NODELIST(REASON)
        9490      work 1b1bc6bc-b17    jalison PD       0:00      1 N/A (ReqNodeNotAvail, UnavailableNodes:rogue[01-02])
        9489      work 1b1bc6bc-b17    jalison PD       0:00      1 N/A (Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions)
        9378      work analysis_sys  reana-cmu  R    5:19:10      1 N/A rogue01
        9379      work analysis_sig  reana-cmu  R    5:19:10      1 N/A rogue01
  
  

• bailing on falcon… runing on cmslpcgpu1…
• fix for mix4 data not being picked up

• fix for new LPC container needed for old CUDA on the gpu node gpus

  ./run_container snakemake --profile software/snakemake/profiles/lpc_gpu --snakefile coffea4bees/workflows/Snakefile_Run3MvD.smk output/Run3_MvD_quadjet_run2_rank0/MvD/train.done output/Run3_MvD_quadjet_run2_rank0/MvD/evaluate.done output/Run3_MvD_quadjet_run2_rank0/MvD/analyze.done --cores 4 --config dataset_name=mixeddata_all_rank0 mode=quadjet_run2 --rerun-triggers mtime --logger snkmt
  

• Now running …
• Mega MvD…
• making pour sessions
• Updating slides…
• submitting training

• Finished first pass at MvD slides
• [X] Do nominal quadjet_run2 FvT

• Making slides…. Claude's Suggestions
• Good progress

• SvB_FeynNet hists are there now !
• Updating pourOvers
• [X] Run FvT with FeynNet scores…

• No SvB_FeynNet hists!
• Debugging
• Rerunning… on cmslpc309
• Crashed … Rerunning… something is locking me out… debugging

• Adding SvB_Feynet
• Making SvB_Feynet Friend Trees
• Setting up with claude
• test script

./run_container source coffea4bees/scripts/SvBFeynNet-friendtree-test.sh

• No need for top_{reconstruction}
• Snakemake workflow:

./run_container snakemake --snakefile coffea4bees/workflows/Snakefile_SvBFeynNet_friendtrees_Run3.smk --profile software/snakemake/profiles/lpc --cores 4 --dry-run --printshellcmds --logger snkmt

• Running… probably should have used tmux…
• Crash b/c output condor logs collisions.. fixed
• Adding an option to flush output… doesn do anything
  • [>>] Revert
• Now running on tmux on cmslpc310
• Remaking histograms with SvB_Feynet… tmux on cmslpc310
• Running…

• Seeing long tail in MvD weights compared to FvT

• Updating the MvD_noMvD (and noFvT) histograms weights to work when scaling from ttbar
• Still see problems with noMvD weights !
• Problem is with data not backgrounds
• Fixed .. issue with jcm being applied to datat.
• Added plot of highMvD
• Looking at a lot of plots…
• [>>] Idea for webapp plot viewer

• Looks like the MvD evaluate didnot write to eos… Running from falcon
• Evaluate is hanging on falcon… Trying on GPU1
• Also hanging on GPU1… now running… finished
• Making plots
• python -i coffea4bees/plots/iPlot.py --combine coffea4bees/Run3_MvD/analysis_MvD.coffea coffea4bees/Run3TriggerSFs/trigWeight_TTbar_wHLT_newJCM.coffea -m coffea4bees/plots/metadata/plotsAll_MvD.yml
• Playing with histograms to reduce memeory … see 4b code base
• no-MvD histograms way off.. need to debug…
• ttbar weights not great…

• Training on Falcon
• ssh into it
• Start a tmux session
• ./run_container classifier
• then source the run script
• finished!

• checking Cutflows… TT and data look good ! data 314250.0 TT: 15137.0 + 9377.1 + 1233.0 = 25747.1 mix4: 3294015.0 (unit) No MvD: 275542.54816881445 (agrees with study mixed datasets ) w/MvD: 194857.0487652345

/ 3294015.0

• Data different in apply_MvD… looks like a change from AE in jet JES
• Will rerun classifier inputs for data
• [X] Nans in threeTag cutflow for data in C4
• Adding no-MvD plots for debugging mixeddata_all
• DONT NEED THE OFFSET IN JCM WHEN TRAINGING!!!
  • It just reads from the lists

• Retraingin…

  In region SR:

In region SB:

• Fails b/c someone ran a job on gpus… killed mine.. he is also running on 2

• Running eval and training… finished
• No NaNs in test file!
• Running full files
• [X]

  Norm off and nJets == 4 bin 0 for mixed..?!

  

• I tihnk the jet multiplcity is wrong for the JCM evaluation!
  • Need to check
    • [X] Study
    • [X] split
    • [X] Fit (The fit was actually OK, b/c the look up with with selected jets)
    • [X] MvD
• Found problem with the JCM… its the +1 offset needed for the mixeddata .. This will likely explain the larger AUC I was seeing in the MvD training
• ok But now seeing…

• Finished .. Evaluated .. Running
• Ran test.. Now running all
• Normalization still fucked up !
• Add plot of noMvD
• printouts of JCM weights in training
• cutflow in SB

• test script now works!
• Running all on condor… no SvB … done
• [ ] FvT and MvD in same job ? … prpobably no .. FvT with data and MvD with mixeddata_all
• [X] Add ttbar from MvD weights
• Added plots and cutflwo from MvD weightfs
• Fixed and issue with mixeddata_all eras not being merged in the histogram axis
• CI failing …
• Rerunning MvD hists
• Seeing nans in MvD…
• Debugging the nans….
• Lots of good work on the NaNs… there are Nans in the input jets.. probably some cock-up in the boosting
• Adding a script to check
• [X] test mixed CI .. then all
• Fixed jet NaNs …
• Script to check classifier inputs for Nans
• Fixing CI
• Remaking the the mixeddata_all dataset … Finished ! … NaNs are gone!
• Make script to output datasets from pico*yml output ./run_container python src/tools/make_dataset_yml.py -i output/mixeddata_make_dataset_Run3_all/picoaod_datasets_mixeddata_Run3_noTT_pz.yml -o coffea4bees/metadata/datasets_HH4b_Run3/mixeddata_all_new.yml -n mixeddata_all
• Documented in the src/tools/README.md
• Running classifier inputs… DONE
• Running Training on gpu1 (tmux)… No more Nans…

• But only see half the number of mixeddata_all events !

  In region SR:

In region SB:

• Need to run study mixed data sets and compare … Looks fine
• Its the ERA config in the common.yml…rerunning …
• Now see

In region SR:

In region SB:

• Figuring out how to apply the MvD weights
• Claude Coded the results
• Test failed … MvD run only on data … not on mixeddata_all
• Rerunning evaluate on gpu1 (in Tmux) … done
• Re-testing… needed another fix for evaluate

• Problems with HCR inputs not aligning…

  root://cmseos.fnal.gov//store/user/jda102/XX4b/Run3ClassiferInputs/mixeddata_all_2023_BPix/HCR_input_mixed_all.chunk8.root
    has     60269
  

• [>>] In evaluate the tmp files are stored locally not in eos
• More info … look like there are some file collisions during HCR_Input making…
• [ ] Script to check that each friend tree is unique
• Rain out of daily claude quota…
• OK I think this is a problem with not haveing eras in mixeddata_all… Update dataset structure to be just like data.yml
• retraining on gpu1 (tmux) .. errors … fixing them with claude
• Training running… Still see the damn NaNs.. Finished
• Running Evaluate… Fixed !

• OK on cmslpcgpu2
• ./run_container classifier
• debugging with Claude
• Training …. In region SR:

In region SB:

• Counts look OK…
• NaNs calling JCM fit… threeTag ttbar ?
• Seeing NaNs in Ghocst batch embedderfs

• loss is being reported as a nan

  * Session Summary
  
  What we built
  
  Implemented the MvD classifier — an alternative to FvT that fits
   mixeddata_all (4b) vs detector data (4b) + ttbar, producing a
   per-event weight MvD = (p_d4 - p_t4) / p_mix4.
  
  New files:
   - src/classifier/config/dataset/HCR/_picoAOD.py — added
     _mixeddata_all() loader + MixedAllBackground class
   - src/classifier/config/dataset/HCR/MvD.py — dataset config (labels
     mix4, t4, d4)
   - src/classifier/config/model/HCR/MvD/baseline.py — loss, ROCs,
     output definition
   - coffea4bees/classifier/config/workflows/HH4b_Run3/MvD/ —
     train.yml, evaluate.yml, run.sh
  
   Bugs fixed during testing
   1. CollisionData.years → CollisionData.eras in _mixeddata_all()
      (metadata keys are era-style e.g. 2022_EE)
   2. _common_selection returns a boolean mask, not a DataFrame —
      needed _mixed_all_selection wrapper
   3. fourTag has ~35k NaN/chunk in mixed_all (3b events, no friend
      tree entry) — fixed fillna(False) in JCM
   4. xW/xbW have 40 NaN/chunk in mixeddata_all friend tree (failed top
      reco) — fixed by _fill_nan preprocessor in MvD.py
   5. NaN loss — caused by xW/xbW NaN propagating through GBN init —
      should be resolved by _fill_nan
  
   Where we left off
  
   The _fill_nan fix was just applied. Training needs to be re-run to
   confirm the loss is no longer NaN and that all GBN embedder stats
   are clean.
  

• NotCanJet_mass is also nan.
• Fixed…. now training without any Nans
• [X] Need to understand where these Nans are coming from…. Boosts numerical instabilities
• Finished training !
• Now running eval … on gpu TMUX!!! … worked well
• Looks like the ROC for MvD is higher than for FvT (!?) … was expecting the opposite
• Ran Eval in Tmux … Need to keep doing this!

• Setting up to make classifier input for mixeddata_4b …
• Claude to write ./run_container source coffea4bees/scripts/classifier-inputs-mixeddata-all-Run3.sh --do-test
• Iterating to debug the output directories
• Then it ran the full production job with out really aksing… Production job completed successfully — 25.6M events processed at ~98k events/s in ~12 minutes via condor. Output files are in output/classifier_inputs_mixeddata_all_Run3/
• Chatting about MvD… setting it up… AMAZING!!!!

• Combied classifier inputs file

  ./run_container python -m src.friendtrees.merge_friend_meta -i coffea4bees/metadata/datasets_HH4b_Run3/classifier_inputs_Run3.json  coffea4bees/metadata/datasets_HH4b_Run3/classifier_inputs_mixeddata_all.json  -o coffea4bees/metadata/datasets_HH4b_Run3/classifier_inputs_MvD_Run3.json
  

• JCM

cp -r output/mixeddata_study/jcm_for_subsampling_Run3_noTT_pz/ coffea4bees/analysis/weights/JCM/Run3_MvD

• Debugging CI

Pass

config=defaultdict(<function processor_config.<locals>.<lambda> at 0x7f5df0351990>, {'isMC': False, 'isPSData': False, 'isMixedData': False, 'isSignal': False,     split_mixed_data.py:56
                           'isRun3': False, 'isSyntheticData': False, 'isSyntheticMC': False, 'isDataForMixed': False, 'isTTForMixed': False, 'cut_on_lumimask': True, 'cut_on_HLT_decision':                        
                           True, 'do_MC_weights': False, 'do_jet_calibration': True, 'do_lepton_jet_cleaning': True, 'override_selected_with_flavor_bit': False, 'use_prestored_btag_SF':                            
                           False, 'do_jet_veto_maps': False})

Fail:

config=defaultdict(<function processor_config.<locals>.<lambda> at 0x7ff8913bd990>, {'isMC': False, 'isPSData': False, 'isMixedData': True, 'isSignal': False,      split_mixed_data.py:56
                           'isRun3': False, 'isSyntheticData': False, 'isSyntheticMC': False, 'isDataForMixed': False, 'isTTForMixed': False, 'cut_on_lumimask': False, 'cut_on_HLT_decision':                       
                           False, 'do_MC_weights': False, 'do_jet_calibration': False, 'do_lepton_jet_cleaning': False, 'override_selected_with_flavor_bit': False, 'use_prestored_btag_SF':                         
                           False, 'do_jet_veto_maps': False})