Run3 Hemisphere Mixing

https://indico.cern.ch/event/1639802/#b-654452-run-2run-3-non-resona

https://indico.cern.ch/event/1655563/

• [X] remake classifier inputs with correct trigger weights ?
• [X] refit JCM
• [X] refit FvT
• [X] Test the weights
• [X] hemispheres subtracting 4b ttbar
  • [X] Make yaml "datatsets" file (by hand)
  • [X] run the study script
• [X] mixing hemispheressubtracting 3b ttbar
  • [X] Update datasets
  • [X] Run Study scipt to get Jet mutliptlicities
• [ ] refit mixed data JCM
• [X] subsample mixed data
• [X] Need the make ttbar 4b PS
• [X] Analyze
• [X] Plots
• [ ] Mail

• [X] I need the JCM for Run3!!!
• [X] Study mixed data
• [X] Split mixed data
• [X] Analysze
• [X] Plots
• [X] Mail

• Running the friendtrees my self…

./run_container snakemake --profile software/snakemake/profiles/lpc --snakefile coffea4bees/workflows/Snakefile_SvB_friendtrees_Run3.smk    coffea4bees/metadata/datasets_HH4b_Run3/SvBfriend_mixeddata_data_rank3_3.json     --config dataset_name=mixeddata_all_rank3_3 reuse_legacy_friends=True     --cores 4 --logger snkmt

• Seems OK …

• Remaking rank0_0 with extra can variables

• Remaking mixed data samples
• [X] Rank0₀
• [X] Rank3₃
• [X] Rank7₇
• Only seeing one jobs at a time run…
• Now Problems with submission, need to redo the bootstrap… re doing….
• [X] Verriffy that previous MvD fit works

• [X] Sent Luka files
• Looks like the mixed samples are indeed smaller than the 3b data
• Half of the mixed events fail pass preselection.. checking why …
  • Its b/c of the 3T+1M … this kill half of the background.

• Checking output…
• [X] Check new plots for better ttbar weights Looks decent
• [X] Compare old and new mixed data / data
  • Looks as expected!

Starting web server on http://localhost:58321

• [X] Mail to luka et al

• Checking output…
• Start the pourover server on LPC
• It was on local host 5000
• open a port to it on my local machine ssh -N -L 5050:localhost:5000 cmslpc307.fnal.gov
• Looks decent … large ttbar weights
• Adding clamping… rerunning the final hists… finished
• [>>] Check new plots for better ttbar weights
• [>>] Compare old and new mixed data / data

• jobs Finished
• Will committ and start traingin
• pushed to git…
• updated code on falcon

./run_container snakemake --profile software/snakemake/profiles/lpc_gpu --snakefile coffea4bees/workflows/Snakefile_Run3MvD.smk
 output/Run3_MvD_quadjet_run2_pt25_rank0/MvD/train.done output/Run3_MvD_quadjet_run2_pt25_rank0/MvD/evaluate.done output/Run3_MvD_quadjet_run2_pt25_rank0/MvD/analyze.done --config
 dataset_name=mixeddata_all_pt25_rank0 mode=quadjet_run2 --cores 4 --rerun-triggers mtime 

Fail… bc lpc_gpu

./run_container snakemake --profile software/snakemake/profiles/falcon --snakefile coffea4bees/workflows/Snakefile_Run3MvD.smk output/Run3_MvD_quadjet_run2_pt25_rank0/MvD/train.done  output/Run3_MvD_quadjet_run2_pt25_rank0/MvD/evaluate.done output/Run3_MvD_quadjet_run2_pt25_rank0/MvD/analyze.done --config dataset_name=mixeddata_all_pt25_rank0 mode=quadjet_run2 tag=pt25 --jobs 4 --rerun-triggers mtime --allowed-rules create_train_yml create_evaluate_yml train evaluate analyze  --logger snkmt

Needed to change profile

• Did training in falcon
• MvD workflwo in claude

• Resubmit jobs…
• Barista

• Merging.
• Starting on the MvD… submitted the workflow.. Running

• make mixing step finished…
• Resuming the rest of the workflow
• More failure… turns out it was b/c of more ak -> ak2 failures… fixing… rerunning…

• Looking into failed workflow.
• Seems like minor transient errors.
• Rerunning…

• Remaking Run3 Hemisphere Mixing with 25 GeV jets in 2023
• Resubmitted the mixing
• Will check with barista … looks good
• fail b/c of memory pressure… rerunning with more GB and more chunks
• Another error… looks like some indexing error… is it akward2 or a pt25 issue ?? debugging …
• Looks like a problem with the nJet partitioning…yes… memory was never the problem.. Rerunning

• Will now remake the mixed dataset

• Running with

  ./run_container snakemake --profile software/snakemake/profiles/lpc --snakefile coffea4bees/workflows/Snakefile_Run3_make_mixeddata.smk --cores 4 --config tag=pt25 hemi_lib=coffea4bees/skimmer/metadata/hemisphere_library_Run3_noTT_pt25.yml hemi_stats_dir=coffea4bees/skimmer/metadata/hemi_statistics_noTT_pt25 default_rank=0 --logger snkmt
  

• Fails b/c of btagging SF, with ttbar events with PNET score-1
• Clamping at 0… now fails with akward v1 ->v2 errors
• Settting up another working to fix barista_console in parallel
• [X] Need to add Run3 mixing CI Handoff note written to: ~/workers/LowerPtFor23Mixing/awkward_migration_handoff.md
• Spun another worker to fix awkward issue… Its running

• Lowering the jet pts in 23.
• Updating code to master
• Migrating threshold in cworker: LowerPtFor23Mixing
• pushed fix for lower pts in 23
• Now … snakemake to make the clusters
• Adding the hemisphere mixing to the mixing snake make mixed_all workflow
• Getting the clustering details from Run3 Hemisphere Mixing
• Make a new snakemake for mixed clustering
• [X] Fix CI counts for lower pt Cut
• Running in tworker makehemis
  • failed… issue with year being eaten
  • failed… not passing fvT Friend trees? … Maybe just needs to be uncommneted in friends ?… fixed…

  • rerunning .. finsihed with some weird errors

    ERROR: no sel jets for tag = 5
    

• Dont we really want (0,1) (1,0) ranks as well as 0=(0,0) and 1=(1,1) ?
• Asking my man claude…implemented! and documented in coffea4bees/hemisphere_mixing/README.md

config:
  use_topk_matching: True
  k_neighbors: 10
  collision_mode: retry
  default_rank: [0, 1]   # pos uses nearest, neg uses 2nd-nearest

• floating the overall normalization parameter, leads to much better fits:

• Setting up mixed_all workflow Snakemake workflow
• Running .. on cmslpc306

• Classifier inputs

  ./run_container snakemake --profile software/snakemake/profiles/lpc --snakefile coffea4bees/workflows/Snakefile_Run3_make_mixeddata.smk --cores 4   --logger snkmt
  

• Error with missing FvT friends… fixed… rerunning
• Looks like the merge step is failing… fixed
• Adding install_dataset step … wiring runner to read mixeddata_allrankN
• Adding study mixeddatasets to snakemake
• Running the study step… Done
  • No duplicate events!
  • Confirmed new output jet branch is there!
• 4b Background from Mixed-data
• Adding Rule for histograms
• Running again on cmslpc306

./run_container snakemake --profile software/snakemake/profiles/lpc --snakefile coffea4bees/workflows/Snakefile_Run3_make_mixeddata.smk all_histograms --config mode=quadjet_run2 --cores 4 --dry-run

• Updating hemi outputs with: Jet_PNetRegPtRawRes

• Adding a flag to veto same hemi events
• Plan to now use further "ranked" hemisphere… NN = rank-0 Next-to-NN = rank-1 ect
• Will now replace hemis with same events
• Implementing… Implemented with option to to run. … merged OK
• Testing with option turned on…

Wierd Mixed event

For run 359998 / event 1639099603 in mixeddata_noTTpz/mixeddata_all2022EE/picoAOD_{mixedv0.chunk5.root} (entry 85561): Original event hemispheres (Old):

• pos: run=359998, event=1639099603, hemId=+1, thrust_phi=2.814

Matched hemispheres (New):

• pos: run=361990, event=42106932, hemId=+1, thrust_phi=1.398, match_dist=0.039
• neg: run=361990, event=42106932, hemId=+1, thrust_phi=1.398,match_dist=0.074

Same source event AND same hemisphereId on both sides → literally the same hemisphere copied twice.

The two kdtree queries (one for pos_hemi, one for neg_hemi) independently hit the same library entry as nearest neighbor — there's no uniqueness constraint in replace_{hemisloadkdTrees} (coffea4bees/hemisphere_mixing/mixing_helpers.py:659) preventing the same library hemisphere from being matched to multiple query hemispheres in the same chunk, let alone to both halves of the same event. The pos/neg subsets even live in different jet-multiplicity bins potentially, but here they were both in the (3 selJet, 2 tagJet, 3 jet) bin and the nearest-neighbor of both query points was the same library hemisphere.

• Setting up "Run-2" candidate selection in Run3 analysis … maybe this is problem with jet multiplicity
• Adding plots of jets passing 40 GeV…

• Finish Slides
• [ ] Plot of the boosts …

• Making slides

• Nominal plots

  python -i coffea4bees/plots/iPlot.py         coffea4bees/Run3_hemisphere_mixing/allRun3Data_wSvB.coffea -m coffea4bees/plots/metadata/plotsAll_ttbarWeights.yml
  

• TTBar Studies

  python -i coffea4bees/plots/iPlot.py --combine  coffea4bees/Run3TriggerSFs/trigWeight_TTbar_wHLT.coffea coffea4bees/Run3TriggerSFs/trigWeight_data.coffea -m coffea4bees/plots/metadata/plotsAll_TTCR_3b.yml
  
  python -i coffea4bees/plots/iPlot.py --combine  coffea4bees/Run3TriggerSFs/trigWeight_TTbar_wHLT.coffea coffea4bees/Run3TriggerSFs/trigWeight_data.coffea -m coffea4bees/plots/metadata/plotsAll_TTCR_4b.yml
  
  plot("trigEm_ll.pt4",region=sum,yscale="linear",cut="passElMu",norm=0,rebin=1,doratio=1,year="Run3")
  plot("trigEm_ll.calojetht",region=sum,yscale="linear",cut="passElMu",norm=0,rebin=1,doratio=1,year="Run3")
  

• Mixed notTT vs mixed

  python -i coffea4bees/plots/iPlot.py --combine coffea4bees/Run3_hemisphere_mixing/allRun3Data_wSvB.coffea coffea4bees/Run3_hemisphere_mixing/histMixedDataRun3.coffea coffea4bees/Run3_hemisphere_mixing/histMixedDataRun3_ttSub.coffea        -m coffea4bees/plots/metadata/plotMixedVsData.yaml 
  
  

• Mixed noTT vs mixed vs Pz

  python -i coffea4bees/plots/iPlot.py --combine coffea4bees/Run3_hemisphere_mixing/allRun3Data_wSvB.coffea coffea4bees/Run3_hemisphere_mixing/histMixedDataRun3.coffea coffea4bees/Run3_hemisphere_mixing/histMixedDataRun3_ttSub.coffea coffea4bees/Run3_hemisphere_mixing/histMixedDataRun3_pz_new.coffea        -m coffea4bees/plots/metadata/plotMixedVsPzVsData.yaml
  
  

python -i coffea4bees/plots/iPlot.py --combine coffea4bees/Run3_hemisphere_mixing/allRun3Data_wSvB.coffea coffea4bees/Run3_hemisphere_mixing/histMixedDataRun3.coffea coffea4bees/Run3_hemisphere_mixing/histMixedDataRun3_ttSub.coffea coffea4bees/Run3_hemisphere_mixing/histMixedDataRun3_pz_new.coffea        -m coffea4bees/plots/metadata/plotMixedVsPzVsData.yaml

• Runing with SvB with out JCM weight.. fails a bunch of times b/c memeory
• Remove non-MA SvB hists … running on cmslpc331

• Making plots python -i coffea4bees/plots/iPlot.py --combine coffea4bees/Run3_hemisphere_mixing/allRun3Data_wSvB.coffea coffea4bees/Run3_hemisphere_mixing/histMixedDataRun3_pz_new.coffea coffea4bees/Run3_hemisphere_mixing/histMixedDataRun3.coffea coffea4bees/Run3_hemisphere_mixing/histMixedDataRun3_noTT_full.coffea -m coffea4bees/plots/metadata/plotsMixedNew.yml

and even better (including the synthetic datasets)

python -i coffea4bees/plots/iPlot.py --combine coffea4bees/Run3_hemisphere_mixing/allRun3Data_wSvB.coffea coffea4bees/Run3_hemisphere_mixing/histMixedDataRun3_pz_new.coffea coffea4bees/Run3_hemisphere_mixing/histMixedDataRun3.coffea coffea4bees/Run3_hemisphere_mixing/histMixedDataRun3_noTT_full.coffea coffea4bees/Run3_hemisphere_mixing/synthetic_data_Run3_2025_SvB.coffea -m coffea4bees/plots/metadata/plotsMixedNew.yml

plot("SvB_MA.ps_zz_fine",region=sum,yscale="log",cut="passPreSel",norm=0,rebin=8,doratio=1,year="Run3",process=["data","mix_v0","mix_noTT_v0","mix_pz_v0","syn_v0"])

• Merging

• Making datasets with pz boosted hemispheres
• Running study study.

• Seeing smaller distances!

  

• However the norm is way off… debugging cutflows … is this a trigger thing?.. no
• Looks like the eta distribution is way off.
• Problem in the transformations ?… yes they were only valid for m~0
• Generalizing for m>0 … remaking datasets
• Looks better … nJets still different… will press on
• Refitting JCM ./run_container python coffea4bees/analysis/jcm_tools/make_jcm_weights.py -o output/mixeddata_study/jcm_for_subsampling_Run3_noTT_pz --combine_input_files -i output/weights_trigger_analysis/trigWeight_TTbar_wHLT.coffea output/weights_trigger_analysis/trigWeight_data.coffea output/mixeddata_study/study_mixed_datasets_all_Run3_noTT_pz.coffea -c passPreSel -r SB --jcm_config coffea4bees/analysis/jcm_tools/metadata/mixeddata_all_config_Run3.yml
• Splitting the mixed data… Actually looks pretty good…
• Running the SvB

• Pz-boosts looks better yet!

  

Trying to add the pZ boost with claude code

• Doing the ttbar subsampling
• Debugging… updating the sub_sampling code…. add a CI ?
• Got the local version running
• committing changes…
• Running all on condor
• [X] datasets
• [X] Compare to ttbar

(below mixed data v1 is just ttbar PSdata)

• Now running on the mixed including the ttbar PSdata
• Making SvB friend trees for data
• Setup script and running ./run_container source coffea4bees/scripts/SvB-friendtree-Run3.sh
• Made friend trees
• compared mixed w/tT and no tT treatment
• add dastaset of mixed_noTT_4b
• Make plots with SvB

• Analyzing the mixed data splits
• [X] Compare with Multijet
• Mixed data normalization too Low !!!
• Problem in analyze… The study data looks ~right … but is inconsistent with the analysze output…
• Check on options?
• mixeddata_all2022EE study : v0 = 603392 / all = 8858538 splits: v0 603392.0 / all 8858538.0 analysze presel /all 603392/603392
• Looking by year 2022_* OK .. .2023_BPix OK … problem in preBPIx
• 2023_preBPix study 639988 / 9720561 splits: 639988.0 / 9720561.0 analysze 136918 / 136918
• Problem in the analyse ? … Missing HLT trigger bits … Not sure whats going on here.
• Fixed when running without cutting on HLT
• pushing to head… fixing CI
• Starting to setup TTbar PS skimming… will finish tomorrow

• Making yml file
• There are now a lot more files because I didnt do the clustering from the 4b skims, but from the full data files
• ./run_container python coffea4bees/hemisphere_mixing/study_hemispheres.py --hemifiles coffea4bees/skimmer/metadata/hemisphere_library_Run3.yml --year 2022_EE 2022_preEE 2023_BPix 2023_preBPix --threshold 1000 --out coffea4bees/hemisphere_mixing/hemi_plots_Run3

• New vs old hemis

  Item	New	Old
  nHemis	1198016	1275974
  time	1m39.098s	0m18.952s

  I will remake with the fourTag skims to run … No I cant b/c the FvT is not evaluated for them

• Getting the mixed data going in the test job… number of changes…
• Now remaking all the mixed data… Finished
• 20,583 events/s total (69615572/3382.2631816864014)
• Ran Study hemisphere script… looks good
• I Dont think I need to refit the JCM b/c now the jet mult should be right… I dont think tihs is true be of 3bTT
• In first pass can compare Multijet with mixed_noTT

• Found that the normalization is off.. need to make the JCM for resample afterall

  ./run_container python coffea4bees/analysis/jcm_tools/make_jcm_weights.py  -o output/mixeddata_study/jcm_for_subsampling_Run3_noTT --combine_input_files  -i output/weights_trigger_analysis/trigWeight_TTbar_wHLT.coffea output/weights_trigger_analysis/trigWeight_data.coffea output/mixeddata_study/study_mixed_datasets_all_Run3_noTT.coffea     -c passPreSel -r SB         --jcm_config coffea4bees/analysis/jcm_tools/metadata/mixeddata_all_config_Run3.yml
  

• split the mixed data

• AK: fix for the prescale issue..works!
• resize errors ! … need passHLT only in ttbar selection not the 3b ttbar selection !
• Running on gpu2 … in TMUX!
• Train runs!
• There is an error in analyze…fixed…string conversion
• Running evaluate…bugs.. now fixed
• Running on data with the new FvT weights… ERRORS…Chat AK: need to move to master
• Update to master.. now evaluate OK
• Now the FvT friend trees work!
• Running all in condor 4GBs … spilling some to disk … should increase… now trying with 5GB… finished
• FvT looks OK
• Running TTbar MC with new JCM to validate trigger weights … finished
• t3 looks good

• t4 looks good

• processor_make_hemi_library.py moving to ttbar subtraction with FvT
• Working…
• Now making all the 4b hemisphere with the ttbar subtraction
• Finsished … results in /store/user/jda102/XX4b/hemiLib_noTT

• Updating JCM to float 4b-tt
• Looks much Better !

• Now what should I actually do witht the TTbar component in the synthetic data ?
• However the d is really pulled… not sure I want to use this after all
• Next steps… see abvoe
• Removing the TT classifer inputs…
• remaking classifier inputs…
  • Running
    • [X] TTTo2L2Nu … one job fils… trying again with more mem… looks worse… trying with less memory…wow! much better… was using too much
    • [X] TTToLep and Had together… Double Wow! def using less mem per job is the way to go here
• Combined outputs to Run3 json file… copied to metadata
• Fitting JCM… "v2" ./run_container python coffea4bees/analysis/jcm_tools/make_jcm_weights.py -o coffea4bees/analysis/weights/JCM/Run3 -c passPreSel -r SB --combine_input_files -i output/weights_trigger_analysis/trigWeight_TTbar_wHLT.coffea output/weights_trigger_analysis/trigWeight_data.coffea -w v2
• Setting up FvT round2
  • Update JCM/ input files / adding passHLT selection to ttbar

  • Running on cmslpcgpu2

    
    > ./run_container classifier 
    Singularity> export WFS="src/classifier/config/workflows/HH4b_Run3/FvT"
    Singularity> source ${WFS}/run.sh 
    
    

  • Seeing strage errors with the TTbar PS…
  • reverted to not use passHLT… still there…
  • Now only the inputs changed…classifier inputs…
  • Seems to be a problem with the presaling
    • Turning for the PS for the moment…now running
    • [X] Try Anikets example source src/classifier/config/workflows/HH4b_2024_v2/FvT/run.sh
    • Same issue!… Turned out to be a problem with change in uproot... fixed later
• Mega work debugging FvT and making inputs

• [X] Test 3b and 4b ttbar wieghts
  • weight 3b data to 3b and 4b ttbar
  • compare to
• [X] Plots with ttbar weights
• [>> ] Cut flow with ttbar weigts
  • Adding option to plots ttbar from data using weights.
• needed to use processor.accumulate to combine plots from differnt process…works!
• Running on all data 41,770 events/s total (69615572/1666.649035692215) ~30min
• Modeling of t3 looks OK but t4 looks off… weird… Not running with correct JCM!
• Re-running… Looks good!

  • python -i coffea4bees/plots/iPlot.py coffea4bees/Run3_hemisphere_mixing/allRun3Data_wTTWeights2.coffea -m coffea4bees/plots/metadata/plotsAll_ttbarWeights.yml

    

• Four tag ttbar modelling now looks good

  

• t3 still off… I Assume its b/c Im running the ttbar MC with the wrong JCM
  • would rerun.. but then I changed the trigger weights treatment in the mean time.
• Will now work on understanding JCM fit… THen redo FvT with update trigger SF and JCM

• Ran all data (evaluating SvB …) 8,764 events/s total (69615572/7943.311356544495) = 2.2 hrs
• [X] Weight data by 3b and 4b ttbar weights in same job

• Updating the dataset paths in the evaluate step

• Running the evaluate step

  [02/07/26 10:02:13] [   main] INFO     Evaluated 3 models in 0:54:11.332619                                                                                                         evaluate.py:103
  [02/07/26 10:08:27] [   main] INFO     Completed 1 analysis in 0:06:13.453446                                                                                                          analyze.py:81
                                  [   main] INFO     Total run time: 1:01:06.192183                                                                                                                            main.py:309
  
  

• Testing the FvT output … looks OK
• Running full job with data and tTbar … fails !
• Running data only
• Looks like the FvT output file only has the results for data
  • They are!… Takes to long to output the 3tag tT… If I want it for 4b TTbar skim
• Test Trigger SF + passHLT bit for MC… works !

• Remade JCM… still not great…

… but better than before

• Probably not worth remaking the classifier inputs until this is significantly better
• [X] Should vett the TTBar 3b and 4b weights

• running the classifier
• Fix the dataset metadata yml file
• Seeing an error on rogue ERROR Failed to read root://cmseos.fnal.gov//store/user/jda102/XX4b/Run3ClassiferInputs/TTToSemiLeptonic_2023_preBPix/HCR_input.chunk111.root confirmed I can read the file "by hand" with root in the combine shell…
• Trying again… same failure… Aniket reports this is a known issue on rogue need to update some software to allow for longer timeout time
• Aniket suggests to try gpu2 on lpc

alias cmslpcgpu2='ssh -XY jda102@cmslpcgpu2.fnal.gov'

• Now getting Error from mass region… Seem like the timeouts are not a problem here.
• Have to change the massregion definitions
• Now Running !!!

https://gitlab.cern.ch/cms-cmu/barista/-/blob/documentation/docs/classifier/backend.md?ref_type=heads

• [X] For ttbar MC we are using the MC efficeincy directly… Marina seems to be cutting on the passHLT and applying the SF
• FvT Tringing finsihed

Training: run 20 epochs in 3:05:44.198942
Training: run 20 epochs in 3:06:44.336339
Training: run 20 epochs in 3:08:07.103907
Finetuning: run 1 epochs in 0:03:11.986272
Saving model to  root://cmseos.fnal.gov//store/user/jda102/HH4b_Run3/classifier/FvT/model_FvT__kfolds_3__offset_1__seed_kfold-FvT-random-0__bbc14666-0563-4cf9-8fe5-19bf64c0f62b.pkl

• The Analyze step worked!
• The evaluate step failed (b/c I didnt update the datasets)….

• Re running the making of hte classifier inputs

  • Starting with TTTo2L2Nu..Did SemiLeptonic …
  • Finished TTbar .. fighting with data… did 2022.. .running 2023… now finished 2023

  ./run_container python -m src.friendtrees.merge_friend_meta -i output/classifier_inputs_Run3/classifier_inputs_*json -o output/classifier_inputs_Run3/classifier_inputs_Run3.json
  

• Looking into fitting FvT
• https://barista.docs.cern.ch/classifier/
• Updating code on falcon
• In barista … running apptainer exec -B .:/srv --nv --pwd /srv docker://chuyuanliu/heptools:ml bash --init-file /entrypoint.sh
• Works !
• running the pyml.py … tab complettion not working … probably problem with folder structure … Aniket shows me fix
• Setting up workflow in barista
• [X] Need JCM

./run_container python coffea4bees/analysis/jcm_tools/make_jcm_weights.py -o coffea4bees/analysis/weights/JCM/Run3/ -c passPreSel -r SB --combine_input_files -i output/weights_trigger_analysis/trigWeight_TTbar.coffea output/analysis_test_Run3/allRun3Data.coffea -w v0

• Try to run

  ./run_container classifier
   Singularity> export WFS="src/classifier/config/workflows/HH4b_Run3"
   source ${WFS}/FvT/run.sh
  

• Think there is a problem with the dataset file… need the old style yamls

• Should we be subtracting 3b ttbar with SvB weights or FvT weights?
  • Probably FvT weights!
• [X] Compare subtraction weights FvT vs SvB for 4b and 3b
• Comparing tt_vsmj weights for SvB and FvT. Look good for 4tag but 3tag is sytematically different!

• Moved ttbar subtraction to use the FvT 3b (option to use 4b)
• Making classifier inputs
  • Tried chunksize: 10000 , was giving 22k jobs now do chunksize: 100000
  • Job failures.. trying data only… still fails…

• Fit JCM .. figure out what I need for DvT training.
• ./run_container python coffea4bees/analysis/jcm_tools/make_jcm_weights.py -o output/mixeddata_study/jcm_for_subsampling_Run3_wTT -c passPreSel -r SB --combine_input_files -i output/weights_trigger_analysis/trigWeight_TTbar.coffea output/analysis_test_Run3/allRun3Data.coffea

• JCM looks off

  

• Should look into plots with no JCM weights
• Starting to update analysis-test-Run3.sh

• merged with head fixes from AE
• Failures in CI
• Testing locally source coffea4bees/scripts/run-local-ci.sh mixeddata-analyze-cutflow
• Fixed!

• Merge Master

  • CI Fails… running locally source coffea4bees/scripts/run-local-ci.sh analysis-cutflow-Run3 works when reverting to old loose…Updating numbers. ./run_container source coffea4bees/scripts/analysis-cutflow-Run3.sh --output-base CI_output cp CI_output/analysis_cutflow_Run3/test_dump_cutflow.yml coffea4bees/analysis/tests/known_Counts_Run3.yml
  • Seeing weird failure in SvB frend tree.. think this is one for AE… sent mail.
  • Looking into CI failure of old mixing. source coffea4bees/scripts/run-local-ci.sh analysis-mixed-cutflow

• Found that I am not processing all the files in datasets yaml … The number of chuncks increased!
• Rerunnign analysis… FIXED !!!!
• Now running with SvB… Done
• Sent mail with prelimiary "v0" mixed datasets
• Running Run3 fourTag data with SvB… Looks DAMN GOOD!!!!

• Time to clean up and push
• Making plots for slides tomorrow
• python -i coffea4bees/plots/iPlot.py --combine coffea4bees/Run3_hemisphere_mixing/histMixedDataRun3_new2.coffea coffea4bees/Run3_hemisphere_mixing/allRun3Data_fourTag.coffea coffea4bees/Run3_hemisphere_mixing/synthetic_data_Run3_2025.coffea -m coffea4bees/plots/metadata/plotMixedVsData.yaml

• Checking new JCM
• Normalization still off in Run3 splitting !
• Looks like splitting command failed… Rerunning ./run_container source coffea4bees/scripts/mixeddata-split-dataset-Run3.sh
• split data JCM still off!
• Looks OK in the study processor
• The JCM from the study processor looks like the correct weights are being applied
• Making separate mixeddatra-split-Run3 dataset

In the skimmer

ixeddata_all_2022_EE::     0:100000 >>>  event.pseudoTagWeight was [0.0383, 0.0383, 0.105, 0.0383, 0.0383, 0.105, 0.0383, 0.135, 0.0383, 0.0732] 

mixeddata_all_2022_EE::     0:100000 >>>  event.pseudoTagWeight is now [0.0406, 0.0406, 0.111, 0.0406, 0.0406, 0.111, 0.0406, 0.111, 0.0406, 0.0774] 

mixeddata_all_2022_EE::     0:100000 >>>  event.pseudoTagWeight was [0.0383, 0.0732, 0.0732, 0.0732, 0.0383, 0.162, 0.105, 0.0732, 0.0383, 0.0383] 

mixeddata_all_2022_EE::     0:100000 >>>  event.pseudoTagWeight is now [0.0406, 0.0774, 0.0774, 0.0774, 0.0406, 0.17, 0.0774, 0.0774, 0.0406, 0.0406] 

mixeddata_all_2022_EE::     0:100000 >>>  event.pseudoTagWeight was [0.0732, 0.0383, 0.105, 0.0383, 0.0732, 0.0383, 0.0383, 0.0383, 0.135, 0.0732] 

mixeddata_all_2022_EE::     0:100000 >>>  event.pseudoTagWeight is now [0.0774, 0.0406, 0.111, 0.0406, 0.0774, 0.0406, 0.0406, 0.0406, 0.142, 0.0774] 

In the study

mixeddata_all_2023_preBPix::     0:100000 >>>  event.pseudoTagWeight was [0.21, 0.0732, 0.0732, 0.0383, 0.105, 0.0383, 0.0732, 0.105, 0.0732, 0.0383] 

mixeddata_all_2023_preBPix::     0:100000 >>>  event.pseudoTagWeight is now [0.196, 0.0774, 0.0774, 0.0406, 0.111, 0.0406, 0.0774, 0.111, 0.0774, 0.0406] 

mixeddata_all_2023_preBPix::     0:100000 >>>  event.pseudoTagWeight was [0.0383, 0.0732, 0.105, 0.0383, 0.0383, 0.0732, 0.0732, 0.105, 0.0732, 0.0383] 

mixeddata_all_2023_preBPix::     0:100000 >>>  event.pseudoTagWeight is now [0.0406, 0.0774, 0.111, 0.0406, 0.0406, 0.0774, 0.0406, 0.111, 0.0774, 0.0406] 

• [ ] Compare splitting what what shoudl be from the study processor …

• [X] Fit new JCM
• [X] Split (v0 - v2)
• [ ] Plots
• [ ] Mails
• [ ] Clean code
• Fitting new JCMf (make a config for mixed data with no TT) ./run_container python coffea4bees/analysis/jcm_tools/make_jcm_weights.py -o output/mixeddata_study/jcm_for_subsampling_Run3 -c passPreSel -r SB --combine_input_files -i output/analysis_test_Run3/allRun3Data.coffea output/mixeddata_study//study_mixed_datasets_all_Run3.coffea -jcm_config coffea4bees/analysis/jcm_tools/metadata/mixeddata_all_config_noTT.yml -m coffea4bees/plots/metadata/plotsJCM_noTT.yml
• Moved to coffea4bees/skimmer/metadata/jetCombinatoricModel_for_mixed_Run3_splitting.txt
• Splitting the mixed data ./run_container source coffea4bees/scripts/mixeddata-split-dataset-Run3.sh
• Adding split datasets to metadata file
• Running analysis (w/o SvB) ./run_container source coffea4bees/scripts/mixeddata-analyze-Run3.sh
• Making plots… Looks like the I used the wrong JCM…
• Resplitting

• Checking new Loose2
• Tightening the LooseWP… Replace the L value with the average of L and M
  • v0 (Loose) Before: 23709032.0 (threeTag passPreSel)
    • v1 (L+M /2) : 8025974.0
    • v2 (2L + M) / 3 10268888.0 => a muQCD of 14.21

• Remaking JCM weights ./run_container python coffea4bees/analysis/jcm_tools/make_jcm_weights.py -o output/analysis_test_Run3/jcm_for_Run3_noTT -c passPreSel -r SB -i output/analysis_test_Run3/allRun3Data.coffea --jcm_config coffea4bees/analysis/jcm_tools/metadata/noTT_config.yml -m coffea4bees/plots/metadata/plotsJCM_noTT.yml

  • Looks decent:

  

• copy to coffea4bees/skimmer/metadata/jetCombinatoricModel_for_mixed_Run3.txt
• Making all mixed data ./run_container source coffea4bees/scripts/mixeddata-make-dataset-Run3-all.sh
• Failed: distributed.scheduler.KilledWorker: Attempted to run task 'HemiMixer-2092627aecaa11fa213e0504d0f56025' on 4 different workers, but all those workers died while running it. The last worker that attempt to run the task was tcp://131.225.207.249:10000. Inspecting worker logs is often a good next step to diagnose fwhat went wrong. For more information see https://distributed.dask.org/en/stable/killed.html.
• Trying locally File "/srv/coffea4bees/hemisphere_mixing/mixing_helpers.py", line 625, in replace_{hemisloadkdTrees} match_dist, match_idx = kd_tree.query(subset_hemispoints, k=1) File "_ckdtree.pyx", line 798, in scipy.spatial._{ckdtree.cKDTree.query}

ValueError: 'x' must be finite, check for nan or inf values

• Finding errors from nans in the hemis being mixed… now filtering them out
• Re-running on condor… still fails
• Trying locally … seems OK
• Will try one era at a time on condor
  • 2023_preBPix… died … Trying locally.. Fails… looks like running out of mem
• Trying condor again with 6GB condor mem… seems OK got a wierd fill missing error… will run again Turns out this was the magic bullet!
• running 2023_BPix… Runs fine… 15,789 events/s total (14224083/900.8916823863983)
• Profiler shows lots used in the if not (hasattr(p4, "px") and hasattr(p4, "py")): calls.
• Commenting out checking again
• Running 2022 in one go…All good 34,265 events/s total (28706572/837.7758021354675)
• Rerunning 2023_preBPix… Now all good 33,315 events/s total (26684917/800.9806606769562)
• Add mixeddata_all to the datasets file datasets_HH4b_Run3_2025_Run3_skims.yml
• Running the study hemispheres… crashed.. b/c trying to run the jet calibration! Hacked with synthetic data flag for now… Runs
• Need to fit the JCM python -i coffea4bees/plots/iPlot.py --combine coffea4bees/Run3_hemisphere_mixing/study_mixed_datasets_all_Run3.coffea coffea4bees/Run3_hemisphere_mixing/allRun3Data_NewLoose2.coffea -m coffea4bees/plots/metadata/plotsMixedNew.yml

• [ ] Draft Mail re:trigger SFs Talked to Sindhu and AE
• Updateing printCutflow tools to understand Run3 skims
• Adding TwoTag to the cutflow
• The Run3 skims are almost all threeTag
• [ ] How do the Run3 yeilds compare to the Run2 Yeilds?
• plotting 4tag nJet vs 3tag plot("selJets_noJCM.n",region="SB",yscale="linear",norm=1,rebin=1,doratio=1,process=["data","Multijet"])
• Will try to fit JCM without ttbar

./run_container python coffea4bees/analysis/jcm_tools/make_jcm_weights.py -o output/analysis_test_Run3/jcm_for_Run3_noTT -c passPreSel -r SB -i output/analysis_test_Run3/allRun3Data.coffea --jcm_config coffea4bees/analysis/jcm_tools/metadata/noTT_config.yml -m coffea4bees/plots/metadata/plotsJCM_noTT.yml

• Need to update loose WP numbers… to have JCM closer to 15
• Tightening the LooseWP… Replace the L value with the average of L and M
  • Before: 23709032.0 (threeTag passPreSel)
    • After: 8025974.0 now about 11x four tag data… should tighten it
• Re running all the Run3 data (condor) … Looks like I am computing SvB…. Should speed it up by turning this off … Killed jobs
• Testing times in test job w/SvB: 295 events/s total (13998/47.519678592681885) no SvB: 465 events/s total (13998/30.101012706756592)
• Rerunning 23,685 events/s total (69615572/2939.2139101028442)
• Loosening the loos to (2xL + T )/ 3
• Rerunning…

• Looking into Run-3 4b skims
• datasets_HH4b_Run3_2025_Run3_skims.yml is the skim after the cutflow challenge should use the data in there.
• Will start with no-TTbar option
• ./run_container source coffea4bees/scripts/analysis-runAll-Run3.sh

• runnong on condor cmslpc318

• Fix issue with local index (needed to specificy axis=1)
• Finding nans in hemisphere library … debugging
• Combined mass is nan for (0,1,1)
• Added filter for nans (was only one hemisphere!) when making the hemistatistics..
• Fixed!
• Mixing all Run3 on condor… runs 5,125 events/s total (2032637/396.6471109390259)
• [X] Run2 mixing now broken
• Fixed logic for which jet branches need to be kept in Run3
• Re-Running Run3 Mixing … done
• Seeing very few 4b mixed events … ITs b/c Im running on the 4b skimms!
• Need full Run3 dasta
• What to do about the ttbar ? Think in hte first pass I should try to ignore it.. Fit no-tt JCM

• Starting!!!!!
• Think I will first do it with out ttbar subtraction.. Should be fast
• Updateing code to master… clean branches
• Run3 Synthetic Data
  • SEtting up Run3 clustering script ./run_container source coffea4bees/scripts/mixeddata-cluster-Run3.sh
• Error with some jet fields… Need to thin … fixed..
• Test job now runs
• Running all 22+23 data ./run_container source coffea4bees/scripts/mixeddata-cluster-Run3-all.sh
• Making hemi_stats ./run_container python coffea4bees/hemisphere_mixing/study_hemispheres.py --hemifiles coffea4bees/skimmer/metadata/hemisphere_library_Run3.yml --year 2022_EE 2022_preEE 2023_BPix 2023_preBPix --threshold 1000 --out coffea4bees/hemisphere_mixing/hemi_plots_Run3
• Starting the mixing … Seeing strange error… TBC