Snakemake for 4b Workflows

• Setting up Run3 nominal snakemake
• Adding training / evalauate
• Running on tmux cmslpcgpu1

./run_container snakemake --snakefile coffea4bees/workflows/Snakefile_Run3.smk all_with_training  --config mode=quadjet_run2 --logger snkmt --cores 4   --rerun-triggers mtime 

In region SR:

In region SB:

• Trainging finished… but errors
• Analyze shows errors… but looks like it ran OK
• Looks like chuyuan updated the container image and this broke access to the GPUs
• Chuyuan pushed fix… put still isnt update on cvmfs
• OK now there … still fail.. this is a cuda problem.
• Looks like we will have to revert… pinged him… he will make a new container for cmslpcgpu

• one make hist job stalled… all condor workers cancelled… killed it and reran
• Done! … now have pourovers:

• Histograms missing SvB hists… probably becaue run_SvB: false in the config
• fixing…
• Running on cmslpc354
• Deleted the coffea hists and histogram config files then ran

./run_container snakemake --snakefile coffea4bees/workflows/Snakefile_classifier_inputs_Run3MvD.smk all_MvD_histograms --config mode=quadjet_run2 --cores 4 --rerun-triggers mtime --logger snkmt

• SvB hists are there in the first output files (pre merge)

• Added SvB friend trees for mixed data
• Running on cmslpc312
• Finished
• Added Merged rule
• Added install rule
• update histograms steps to use the merged SvB friends
• Struggling to get the right snakemake command
• .. OK got it going … Running on cmslpc312

• Adding Plots with MvD… on cmslpc311

./run_container snakemake  --snakefile coffea4bees/workflows/Snakefile_classifier_inputs_Run3MvD.smk all_with_training  --cores 4  --config mode="quadjet_run2"    --logger snkmt         --dry-run -p

• Works out of the box!

• Adding plots no JCM
• [X] EOS check
• [X] Classifier training…
• Still on cmslpc365
• Running plots with no JCM
• [X] Classifier setup
• [X] Update config to allow JCM but not MvD

• On LCP

  ./run_container snakemake  --snakefile coffea4bees/workflows/Snakefile_classifier_inputs_Run3MvD.smk  --cores 4  --config mode="quadjet_run2"    --logger snkmt    --until install_JCM install_classifier_inputs --dry-run
  

• Now trying the tracining on on LPC… gpu1

  ./run_container snakemake  --snakefile coffea4bees/workflows/Snakefile_classifier_inputs_Run3MvD.smk  all_with_training  --config mode="quadjet_run2" --cores 1   --logger snkmt 
  

  """
Traceback (most recent call last):
  File "/opt/conda/envs/hep/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/uscms_data/d3/jda102/HH4b/Run3/barista/src/classifier/config/main/train.py", line 25, in __call__
    return trainer(device=self._device, dataset=self._dataset)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/uscms_data/d3/jda102/HH4b/Run3/barista/src/classifier/ml/training.py", line 239, in train
    self.device = device.get(self.min_memory)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/uscms_data/d3/jda102/HH4b/Run3/barista/src/classifier/process/device.py", line 77, in get
    raise ValueError("No valid device found")
ValueError: No valid device found
"""

• Fixed with container call
• Running training… see some errors…turns out not important
• Evalate runs! … also errors but they are not important!
• Analysis crashed b/c of proxy… getting it working

• Ok back on cmslpc365

• Trying if statements in Snakemake… Works!

  ./run_container snakemake  --snakefile coffea4bees/workflows/Snakefile_classifier_inputs_Run3MvD.smk  --cores 3 --until merge_histograms  --config mode="quadjet_run2"    --logger snkmt 
  

• Fixing the merging… DONE !
• [X] Add JCM step
• Merging

• JCM tst command:

  ./run_container python coffea4bees/analysis/jcm_tools/make_jcm_weights.py  -o test/jcm_for_SM   -i output/Run3_MvD_quadjet_run2/histAll_Run3MvD_quadjet_run2.coffea       -r SB         --jcm_config coffea4bees/analysis/jcm_tools/metadata/mixeddata_all_config_Run3.yml
  

  • Some passPreSel fixin Better…

    

• Got JCM in snakemake
• Planning Other Steps… see above
• Running the classifier inputs
• Trying to write out dask_logs…

• 4b code base
• First aim is classifier inputs: data / mixed_data and TTbar
• Plots noMvD independent of classifier inputs To validate the rule graph (dry run):

./run_container snakemake --snakefile coffea4bees/workflows/Snakefile_classifier_inputs_Run3MvD.smk --cores 4 --dry-run                                                                                                            

To run a single year/dataset subset:

./run_container snakemake --snakefile coffea4bees/workflows/Snakefile_classifier_inputs_Run3MvD.smk --config years='["2022_EE"]' ttbar_datasets='["TTToSemiLeptonic"]' --cores 4

• [X] Make plots to fit the JCM
• 4b Background from Mixed-data
• Running… on cmslpc349 in tmux

./run_container snakemake --snakefile coffea4bees/workflows/Snakefile_classifier_inputs_Run3MvD.smk  --cores 4 --until merge_histograms  --config label="_quadjet_run2" classifier_config="coffea4bees/analysis/metadata/HH4b_classifier_inputs_Run3_quadjet_run2.yml" histogram_config="coffea4bees/analysis/metadata/HH4b_run_fastTopReco_Run3_quadjet_run2.yml"   --logger snkmt 

• To run the snakemake console
  • add in snakemake cmd: --logger snkmt
  • Then out of tumx: pixi run snkmt console
• TTbar jobs failing with btagging SF … debugging
• Need to update when Run3:

correction_type="particleNet_shape",

• on cmslpc365
• Adding histogram_config option to MvD snakeake
• Submitting… No space on device error ! … seems to be /tmp. .. moving back to 349
• Changed to output tar files local.. Running
  • [X] Check this with AE… clean up later
• Monitoring with ~/.pixi/bin/pixi run snkmt console
• Jobs fail b/c trying to use the same condor port… hack to 0
  • [X] Check with AE re:clashing ports
• Rerunning snake make .. only runs the jobs that did not suceed NIce!
• Now merging failing … debug later