Adding MCH aQC for MC

[lmassacr]

Adding to DPG scripts the workflows for the aQC of MCH and MCH+MID objects

[github-actions]

REQUEST FOR PRODUCTION RELEASES:
To request your PR to be included in production software, please add the corresponding labels called "async-" to your PR. Add the labels directly (if you have the permissions) or add a comment of the form (note that labels are separated by a ",")

+async-label <label1>, <label2>, !<label3> ...

This will add <label1> and <label2> and removes <label3>.

The following labels are available
async-2023-pbpb-apass4
async-2023-pp-apass4
async-2024-pp-apass1
async-2022-pp-apass7
async-2024-pp-cpass0
async-2024-PbPb-cpass0
async-2024-PbPb-apass1
async-2024-ppRef-apass1

[chiarazampolli]

Hello @JianLIUhep , @aferrero2707

Can you check this one?

Chiara

[JianLIUhep]

Hi @lmassacr, is MC/config/QC/json/mftmchmid-tracks-task.json needed? I do not see it is included in any workflow you introduced in this PR.

[lmassacr]

Hi @JianLIUhep,

Indeed I anticipated a next PR to come by putting already the json file. I still need indeed to add the workflow for MCH+MFT and MCH+MFT+MID tracks for the MC aQC. The corresponding modifications of the dpg scripts is work in progress. My local tests are currently failing.

Regarding this PR, I see that I have an error in the automatic checks (which I didn't get locally). This is related to the fact the the QC reads some intermediate .root files produced by the muon reco, for which it seems the file is eitheir corrupted or an array was not properly filled.
I was wondering if this could be due to the fact that only few simulated events are produced by the test and the array is empty. I am not sure how to add a protection in the dpg script against this problem. This is actually not directly a feature of the QC code I commited.
@aferrero2707 do you have some suggestions?

Cheers,
Laure

[lmassacr]

Hi @JianLIUhep and @aferrero2707 , I am pinging you again regarding this PR if you have suggestions on how to proceed.
Thanks,
Laure

[JianLIUhep]

Hi @lmassacr, I saw some segmentation and bus errors from other tasks before the mch task crash. Maybe try to make a dummy commit to retrigger the test.
Thanks.

[aferrero2707]

@JianLIUhep @chiarazampolli @catalinristea @alcaliva @lmassacr I am getting the same kind of error locally, using O2DPG master from yesterday.

The error in the CI from the full build log:

==> START BLOCK: Test running workflow with AnalysisQC <==
Test 83: Running AnalysisQC CLI�[0;32m -> PASSED�[0m
Test 84: /O2DPG/MC/bin/tests/wf_test_pp.shERROR for workflows execution and AnalysisQC.
�[0;31mError found in log /sw/BUILD/55f11d850f4d576fb51bf2171d3426090b2e6e97/O2DPG-sim-tests/o2dpg-sim_tests/o2dpg_tests/workflows_analysisqc/84_wf_test_pp.sh_dir/Analysis/MergedAnalyses/Analysis_MergedAnalyses.log�[0m
5343-[21865:track-propagation]: [07:03:02][INFO] MagneticField::Print: Uses Sol30_Dip6_Hole  of /sw/slc9_x86-64/O2/daily-20250201-0000-local1/share/Common/maps/mfchebKGI_sym.root
5344-[21865:track-propagation]: Info in <TGeoGlobalMagField::SetField>: Global magnetic field set to <MagneticFieldMap>
5345-[21865:track-propagation]: Info in <TGeoGlobalMagField::Lock>: Global magnetic field <MagneticFieldMap> is now locked
5346-[21865:track-propagation]: [07:03:02][INFO] Loaded 440 params from $(O2_ROOT)/share/Common/maps/sol5k.txt
5347-[21865:track-propagation]: [07:03:02][INFO] ccdb reads http://alice-ccdb.cern.ch/GLO/Calib/MeanVertex/1546300800000/9c31089f-2c77-11ed-ac0b-2a010e0a0b16 for 1550600800022 (retrieve from snapshot, agent_id: alimetal03.cern.ch-1738389781-fZrhzt), 
5348-[21865:track-propagation]: [07:03:02][INFO] CCDBManager summary: 3 queries, 21,442,560 bytes for 3 objects, 3 good fetches (and 0 failed ones) in 328 ms, instance: alimetal03.cern.ch-1738389781-fZrhzt
5349-[21865:track-propagation]: [07:03:02][INFO] CCDB cache miss/hit/failures
5350-[21865:track-propagation]: [07:03:02][INFO]   GLO/Calib/MeanVertex: 1/0/0 (2048-2048 bytes)
5351-[21865:track-propagation]: [07:03:02][INFO]   GLO/Config/GRPMagField: 1/0/0 (1024-1024 bytes)
5352-[21865:track-propagation]: [07:03:02][INFO]   GLO/Param/MatLUT: 1/0/0 (21439488-21439488 bytes)
5353:[21867:bc-selection-task]: *** Program crashed (Segmentation fault)
5354-[21867:bc-selection-task]: Backtrace by DPL:
5355-[21867:bc-selection-task]: Executable is /sw/slc9_x86-64/O2Physics/daily-20250201-0000-local1/bin/o2-analysis-event-selection
5356-[21867:bc-selection-task]:     /lib64/libc.so.6:     ?? ??:0
5357-[21865:track-propagation]: [07:03:02][INFO] Sending end-of-stream message to channel from_track-propagation_to_bc-selection-task
5358-[21865:track-propagation]: [07:03:02][INFO] Sending end-of-stream message to channel from_track-propagation_to_vertexingfwd
5359-[21865:track-propagation]: [07:03:02][INFO] Sending end-of-stream message to channel from_track-propagation_to_tof-signal
5360-[21865:track-propagation]: [07:03:02][INFO] Sending end-of-stream message to channel from_track-propagation_to_access-mc-truth
5361-[21865:track-propagation]: [07:03:02][INFO] Sending end-of-stream message to channel from_track-propagation_to_track-selection
5362-[21865:track-propagation]: [07:03:02][INFO] Sending end-of-stream message to channel from_track-propagation_to_qa-event-track
5363-[21865:track-propagation]: [07:03:02][INFO] Sending end-of-stream message to channel from_track-propagation_to_lambdakzero-builder

I did my local test with O2DPG master, using this command (the echo is needed once to modify one file and trigger the tests):

export O2DPG_TEST_REPO_DIR=~/Workflows/Software/O2DPG-test 
echo " " >> ${O2DPG_TEST_REPO_DIR}/MC/config/QC/json/mft-tracks-mc.json
${O2DPG_TEST_REPO_DIR}/test/run_workflow_tests.sh

with the sime kind of failure in bc-selection-task:

==> START BLOCK: Test running workflow with AnalysisQC <==
Test 1: Running AnalysisQC CLIESC[0;32m -> PASSEDESC[0m
Test 2: /home/flp/Workflows/Software/O2DPG-test/MC/bin/tests/wf_test_pp.shERROR for workflows execution and AnalysisQC.
ESC[0;31mError found in log /home/flp/Workflows/O2DPG-test/o2dpg_tests/workflows_analysisqc/2_wf_test_pp.sh_dir/Analysis/MergedAnalyses/Analysis_MergedAnalyses.logESC[0m
5752-[119286:track-propagation]: [20:17:13][INFO] MagneticField::Print: Uses Sol30_Dip6_Hole  of /home/flp/Workflows/Software/alice/sw/slc7_x86-64/O2/mrrtf-226-improved-time-clustering-local2/share/Common/maps/mfchebKGI_sym.root
5753-[119286:track-propagation]: Info in <TGeoGlobalMagField::SetField>: Global magnetic field set to <MagneticFieldMap>
5754-[119286:track-propagation]: Info in <TGeoGlobalMagField::Lock>: Global magnetic field <MagneticFieldMap> is now locked
5755-[119286:track-propagation]: [20:17:13][INFO] Loaded 440 params from $(O2_ROOT)/share/Common/maps/sol5k.txt
5756-[119286:track-propagation]: [20:17:13][INFO] ccdb reads http://alice-ccdb.cern.ch/GLO/Calib/MeanVertex/1546300800000/9c31089f-2c77-11ed-ac0b-2a010e0a0b16 for 1550600800022 (retrieve from snapshot, agent_id: mchflp1-1738351028-8SjOLO), 
5757-[119286:track-propagation]: [20:17:13][INFO] CCDBManager summary: 3 queries, 21,442,560 bytes for 3 objects, 3 good fetches (and 0 failed ones) in 575 ms, instance: mchflp1-1738351028-8SjOLO
5758-[119286:track-propagation]: [20:17:13][INFO] CCDB cache miss/hit/failures
5759-[119286:track-propagation]: [20:17:13][INFO]   GLO/Calib/MeanVertex: 1/0/0 (2048-2048 bytes)
5760-[119286:track-propagation]: [20:17:13][INFO]   GLO/Config/GRPMagField: 1/0/0 (1024-1024 bytes)
5761-[119286:track-propagation]: [20:17:13][INFO]   GLO/Param/MatLUT: 1/0/0 (21439488-21439488 bytes)
5762:[119288:bc-selection-task]: *** Program crashed (Segmentation fault)
5763-[119288:bc-selection-task]: Backtrace by DPL:
5764-[119288:bc-selection-task]: Executable is /home/flp/Workflows/Software/alice/sw/slc7_x86-64/O2Physics/master-local1/bin/o2-analysis-event-selection
5765-[119287:access-mc-data]: [20:17:13][INFO] Sending end-of-stream message to channel from_access-mc-data_to_McParticles
5766-[119287:access-mc-data]: [20:17:13][INFO] Sending end-of-stream message to channel from_access-mc-data_to_check-mc-particles-indices-grouped
5767-[119287:access-mc-data]: [20:17:13][INFO] Sending end-of-stream message to channel from_access-mc-data_to_internal-dpl-aod-global-analysis-file-sink
5768-[119288:bc-selection-task]:     /lib64/libc.so.6:     ?? ??:0
5769-[119286:track-propagation]: [20:17:13][INFO] Sending end-of-stream message to channel from_track-propagation_to_bc-selection-task
5770-[119286:track-propagation]: [20:17:13][INFO] Sending end-of-stream message to channel from_track-propagation_to_vertexingfwd
5771-[119286:track-propagation]: [20:17:13][INFO] Sending end-of-stream message to channel from_track-propagation_to_tof-signal
5772-[119286:track-propagation]: [20:17:13][INFO] Sending end-of-stream message to channel from_track-propagation_to_access-mc-truth

[lmassacr]

Hi @aferrero2707,
Thanks for your check. So this confirm the failure is unrelated to my commit (ie. you didn't pick the changes from my PR?).
Can we therefore go ahead with the merging of this PR? or do you want me still to do a test here, commenting my changes?

[alcaliva]

@sawenzel, the crash in the testing is unrelated to the code. Could you force merge this PR?

[sawenzel]

@alcaliva : You are admin. You should also be able to merge.

[sawenzel]

The failing test is due to a broken TPC digitization which was fixed. I would be in favour of waiting for another CI iteration to see that the development here is good.

[lmassacr]

Hi @sawenzel ,

I just made a dummy commit to retrigger the CI. As there were some crashes in the MCH tasks in the first attempt (although maybe related to previous crash in the chain unrelated to this code), it is better to retest.

Cheers,
Laure

[sawenzel]

@lmassacr : Thanks... but the dummy commits are no longer necessary. I had already retriggered the CI in the github actions tab.

[lmassacr]

Hi @sawenzel ,
Sorry for the interference, I didn't know about this. I see that the checks is in now in status "skipped, no relevant changes". Do you need to retrigger it?

[aferrero2707]

Hello @sawenzel! The CI failure seems again to be completely un-related (6:[FATAL] Alien Token Check failed). Could you maybe re-trigger the CI, to see if we can manage to get it fully green?
Thanks a lot!

[jackal1-66]

Hello @sawenzel! The CI failure seems again to be completely un-related (6:[FATAL] Alien Token Check failed). Could you maybe re-trigger the CI, to see if we can manage to get it fully green? Thanks a lot!

Hello @aferrero2707, I just restarted the CI.

[jackal1-66]

The CI was giving an error related to the alien token, however it was refreshed last week in the machine. I just restarted the test to see if it was just a case.

[lmassacr]

Hi @jackal1-66,
The error related to JALIEN seems to persist.

[jackal1-66]

Hi @jackal1-66, The error related to JALIEN seems to persist.

Pinged the machine experts

[jackal1-66]

@lmassacr

287-[INFO]  - Device mch-cluster-reader: pid 17343 (exit 0)
288-[ERROR]  - Device mch-tracks-reader0: pid 17344 (exit 128)
289-[INFO]    - First error: [WARN] MCH global clusters do not support MC lables, disabling
290-[INFO]    - Last error:     o2-global-track-cluster-reader() [0x404c85]:     _start at ??:?
291-[INFO]  - Device internal-dpl-ccdb-backend: pid 17345 (exit 0)
292-[INFO]  - Device qc-task-MCH-Tracks: pid 17346 (exit 0)
293-[INFO]  - Device qc-root-file-sink: pid 17347 (exit 0)
294-[INFO]  - Device internal-dpl-injected-dummy-sink: pid 17348 (exit 0)
295-[INFO] Dumping used configuration in dpl-config.json
296-[ERROR] SEVERE: Device mch-tracks-reader0 (17344) returned with 128

The new failure seems to be legit, can you take a look?

[lmassacr]

Hi @jackal1-66 ,

Indeed this one is on this PR. I have to investigate because I don't have the issue locally.
@aferrero2707 just in case you have some ideas regarding the workflow and json I used :
171-[17344:mch-tracks-reader0]: Error in TFile::ReadBuffer: error reading all requested bytes from file mchtracks.root, got 224 of 300
172-[17344:mch-tracks-reader0]: Error in TFile::Init: mchtracks.root failed to read the file type data.
173-[17344:mch-tracks-reader0]: Error in TFile::ReadBuffer: error reading all requested bytes from file mchtracks.root, got 224 of 300
174-[17344:mch-tracks-reader0]: Error in TFile::Init: mchtracks.root failed to read the file type data.
175:[17344:mch-tracks-reader0]: [13:42:18][ERROR] Exception caught while in Init: can not find branch trackrofs. Exiting with 1.

The intermediate root files seem somehow corrupted. And there is a problem with the trackrofs objects.

[aferrero2707]

@lmassacr @jackal1-66 @sawenzel I am not 100% sure that the last error is really related to the changes in this PR.

The CI fails during test 84 (/O2DPG/MC/bin/tests/wf_test_pp.sh), and the first failure I see is from the bc-selection-task:

5346:[21739:bc-selection-task]: *** Program crashed (Segmentation fault)
5347-[21739:bc-selection-task]: Backtrace by DPL:
5348-[21739:bc-selection-task]: Executable is /sw/slc9_x86-64/O2Physics/daily-20250228-0000-local1/bin/o2-analysis-event-selection
5349-[21739:bc-selection-task]:     /lib64/libc.so.6:     ?? ??:0

Could it be that due to that failure the intermediate ROOT files are not closed properly?

Note that I have been able to reproduce the crash of bc-selection-task also locally (see #1830 (comment)).

[jackal1-66]

@aferrero2707 I checked on the current cvmfs build and I do see the same error when checking test 84. Hence this looks definitely unrelated to this PR.

[lmassacr]

Hello @jackal1-66,
Thanks for confirming. I discussed with @aferrero2707 and we would propose to merge this PR and have a small test production to test if everything is fine. (I guess we can still disable the MCH QC from the global prod in case of issue).
Would it be fine from DPG side?

[sawenzel]

@lmassacr : I think the main error in the CI logs of O2fst/o2 is genuine and related to this PR. The MCHTracksTaskQC_local1 task fails with this PR:

[27540:internal-dpl-injected-dummy-sink]: [15:09:32][INFO] Correctly handshaken websocket connection.
[27530:mch-cluster-reader]: [15:09:32][STATE] IDLE ---> INITIALIZING DEVICE
[27532:mch-tracks-reader0]: Error in <TFile::ReadBuffer>: error reading all requested bytes from file mchtracks.root, got 224 of 300
[27532:mch-tracks-reader0]: Error in <TFile::Init>: mchtracks.root failed to read the file type data.
[27532:mch-tracks-reader0]: Error in <TFile::ReadBuffer>: error reading all requested bytes from file mchtracks.root, got 224 of 300
[27532:mch-tracks-reader0]: Error in <TFile::Init>: mchtracks.root failed to read the file type data.
[27532:mch-tracks-reader0]: [15:09:32][ERROR] Exception caught while in Init: can not find branch trackrofs. Exiting with 1.
[27532:mch-tracks-reader0]: Executable is /data/aliperf/aliperf_workspace/nightly-tests/software/sw/slc7_x86-64/O2/dev-local13/bin/o2-global-track-cluster-reader
[

Here is what I did:

# go to some workspace
cd /tmp/mytest
# create a MC test workflow
bash ${O2DPG_ROOT}/MC/bin/tests/wf_test_pp.sh
# execute all QC tasks
${O2DPG_ROOT}/MC/bin/o2dpg_workflow_runner.py -f workflow.json --target-labels QC

It could be that the error occurs only randomly but it looks as if the mchtracks.root file is either corrupted or incomplete.

[lmassacr]

Hi @sawenzel,

Thanks for your inputs. I am making locally the same tests as you. Although the workflow already crashes because of ft0fV0emcctp_digi, then tpcclusterpart1_1 , then tpcreco_2, mftDigitsQC0_local2 (see for instance attachment)
. I have to launch several time the commands such that it can go through at some point. So I wonder if you see also those intermediate crash and if this can sometime contribute to corrupt the intermediate muon files. I was trying to run only the reco for MCH/MID, but the option to pick specific detectors seems depreciated.

After I managed to go through the reco, the QC part didn't fail (except for mftDigitsQC0_local2 in this specific example), the muon QC goes through and I have the outputs produced (see attachment) and filled with 1 track. I have attached the log of MCHTracksTaskQC_local1 and MCHTracksTaskQC_local2
MCHTracksTaskQC_local1.log
MCHTracksTaskQC_local2.log

[sawenzel]

@lmassacr : What platform are you on? As usual you would need to inspect the relevant log file, such as ft0fV0emcctp_digi_1.log to get information about errors. I am not aware of any production issues in these tasks but one never can be sure.

[lmassacr]

Hi @sawenzel,
I am on MAC OS Sequoia. On which platform did you manage to reproduce the MCH QC task crash?
I have attached the log for ft0fv0emcctp_digi for information (a segmentation fault in the EMCAL digitizer)
ft0fv0emcctp_digi_2.log
although in my case it doesn't cause any problem to the MCH QC later on.
Thanks for your help,

[lmassacr]

Hi @sawenzel, @aferrero2707,

I am making now some tests on linux. I am able to reproduce the crash, though it is yet not clear to me what happens.
If I open the mchtracks.root file, the file looks okay, the trackrofs branch is there and accessible.
If I go in the tf1 directory where the faulty file is, and if I run again the command o2-global-track-cluster-reader --track-types "MCH" --cluster-types "MCH", everything runs fine, see below :
[248923:internal-dpl-clock]: [09:56:36][STATE] INITIALIZED ---> BINDING
[248923:internal-dpl-clock]: [09:56:36][STATE] BINDING ---> BOUND
[248923:internal-dpl-clock]: [09:56:36][STATE] BOUND ---> CONNECTING
[248923:internal-dpl-clock]: [09:56:36][STATE] CONNECTING ---> DEVICE READY
[248923:internal-dpl-clock]: [09:56:36][STATE] DEVICE READY ---> INITIALIZING TASK
[248923:internal-dpl-clock]: [09:56:36][STATE] INITIALIZING TASK ---> READY
[248923:internal-dpl-clock]: [09:56:36][STATE] READY ---> RUNNING
[248923:internal-dpl-clock]: [09:56:36][INFO] fair::mq::Device running...
[248923:internal-dpl-clock]: [09:56:36][INFO] LHCPeriod is not available, using current month MAR
[248923:internal-dpl-clock]: [09:56:36][INFO] Correctly handshaken websocket connection.
[248924:mch-tracks-reader0]: [09:56:37][INFO] branch set up: trackrofs
[248924:mch-tracks-reader0]: [09:56:37][INFO] branch set up: tracks
[248924:mch-tracks-reader0]: [09:56:37][INFO] branch set up: trackclusters
[248924:mch-tracks-reader0]: [09:56:37][INFO] branch set up: tracklabels
[248924:mch-tracks-reader0]: [09:56:37][STATE] INITIALIZING DEVICE ---> INITIALIZED
[248924:mch-tracks-reader0]: [09:56:37][STATE] INITIALIZED ---> BINDING
[248924:mch-tracks-reader0]: [09:56:37][STATE] BINDING ---> BOUND
[248924:mch-tracks-reader0]: [09:56:37][STATE] BOUND ---> CONNECTING
[248924:mch-tracks-reader0]: [09:56:37][STATE] CONNECTING ---> DEVICE READY
[248924:mch-tracks-reader0]: [09:56:37][STATE] DEVICE READY ---> INITIALIZING TASK
[248924:mch-tracks-reader0]: [09:56:37][STATE] INITIALIZING TASK ---> READY
[248924:mch-tracks-reader0]: [09:56:37][STATE] READY ---> RUNNING
[248924:mch-tracks-reader0]: [09:56:37][INFO] fair::mq::Device running...
[248924:mch-tracks-reader0]: [09:56:37][INFO] LHCPeriod is not available, using current month MAR
[248924:mch-tracks-reader0]: [09:56:37][INFO] MCH 5 ROFS
[248924:mch-tracks-reader0]: [09:56:37][INFO] MCH 3 TRACKS
[248924:mch-tracks-reader0]: [09:56:37][INFO] MCH 32 CLUSTERS
[248924:mch-tracks-reader0]: [09:56:37][INFO] MCH 3 LABELS
[248925:mch-cluster-reader]: [09:56:37][INFO] branch set up: clusters
[248925:mch-cluster-reader]: [09:56:37][INFO] branch set up: clusterrofs
[248925:mch-cluster-reader]: [09:56:37][INFO] branch set up: clusterdigits

In that case, I don't specify a json file.

[aferrero2707]

@lmassacr @sawenzel I think I have found the issue... it is in the command here:

        addQCPerTF(taskName='MCHRecoTaskQC',
                needs=[MCHRECOtask['name']],
                readerCommand='o2-mch-reco-workflow',
                configFilePath='json://${O2DPG_ROOT}/MC/config/QC/json/mch-reco-task.json')

The o2-mch-reco-workflow task re-created the already existing mchtracks.root file, and the other commands that read it in parallel then find a corrupted root file.

The solution that worked for me was to add --disable-root-output to the MCH reco task:

        addQCPerTF(taskName='MCHRecoTaskQC',
                needs=[MCHRECOtask['name']],
                readerCommand='o2-mch-reco-workflow --disable-root-output',
                configFilePath='json://${O2DPG_ROOT}/MC/config/QC/json/mch-reco-task.json')

[lmassacr]

Thanks @aferrero2707 for the help. As I have upgraded to Sequoia, I will close this PR and open a new one with the fix, and with the additional workflows for MFT-MCH, and MFT-MCH-MID