Skip to content

Improve cuda gencode flags #321

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
agirault wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Improve cuda gencode flags #321

agirault wants to merge 4 commits into from
+41 −44

Conversation

@agirault
Copy link

@agirault agirault commented Mar 31, 2025 •
edited
Loading

Context

  • The current cuda SASS/PTX list is hardcoded manually based on a versioning heuristic that is error-prone. Case in point:
    • blackwell sm 10.1 is missing
    • blackwell sm 10.0 is supported after 12.8, not 12.6. 
      make[1]: Entering directory '***/gdrcopy/tests'
/usr/local/cuda/bin/nvcc -o pplat.o -c pplat.cu -lcuda -lpthread -ldl -lgdrapi -I /usr/local/cuda/include -I ../include -I ../src -I /usr/local/cuda/include  -gencode arch=compute_60,code=compute_60 -gencode arch=compute_61,code=compute_61 -gencode arch=compute_62,code=compute_62 -gencode arch=compute_70,code=compute_70 -gencode arch=compute_72,code=compute_72 -gencode arch=compute_75,code=compute_75 -gencode arch=compute_80,code=compute_80 -gencode arch=compute_86,code=compute_86 -gencode arch=compute_87,code=compute_87 -gencode arch=compute_90,code=compute_90 -gencode arch=compute_100,code=compute_100 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_62,code=sm_62 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_72,code=sm_72 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_87,code=sm_87 -gencode arch=compute_90,code=sm_90 -gencode arch=compute_100,code=sm_100
nvcc fatal   : Unsupported gpu architecture 'compute_100'
make[1]: *** [Makefile:54: pplat.o] Error 1
  • full arch list are used for code (SASS) and compute (PTX). For PTX, only latest is needed.

Changes

  • Use sm list from nvcc --list-gpu-code directly when available
  • Fix blackwell sm list and version compatibility
  • Consolidate compute & sm list in a single variable
  • Only build PTX for last supported arch

agirault added 4 commits March 31, 2025 12:58
@agirault
build: use sm list from nvcc
e6ea6db 
Signed-off-by: Alexis Girault <agirault@nvidia.com>
@agirault
build: correct blackwell sm use
72fa69c 
- 10.0 is from CTK 12.8+
- 10.1 was missing

Signed-off-by: Alexis Girault <agirault@nvidia.com>
@agirault
build: consolidate arch lists for ptx and sass
c521788 
The list was the same

Signed-off-by: Alexis Girault <agirault@nvidia.com>
@agirault
build: only build ptx for last arch
d1fde74 
Signed-off-by: Alexis Girault <agirault@nvidia.com>
drossetti
drossetti reviewed Apr 3, 2025
View reviewed changes
scripts/get_cuda_gencode.sh
COMPUTE_LIST="$COMPUTE_LIST 120"
SM_LIST="$SM_LIST 120"
# Add Blackwell (10.0, 10.1, 12.0) if CUDA >= 12.8
if [ "$CUDA_VERSION_MAJOR" -ge 12 ] && [ "$CUDA_VERSION_MINOR" -ge 8 ]; then
Copy link
Member

@drossetti drossetti Apr 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this check work for CUDA 13.0?

Copy link
Author

@agirault agirault May 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

scripts/get_cuda_gencode.sh
fi

# Add Ada Lovelace (8.9) if CUDA >= 11.8
if [ "$CUDA_VERSION_MAJOR" -ge 11 ] && [ "$CUDA_VERSION_MINOR" -ge 8 ]; then
Copy link
Member

@drossetti drossetti Apr 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this work on CUDA 12.0 ?

Copy link
Author

@agirault agirault May 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I doubt it would. wasn't in the scope of this MR but I can edit these checks to check major first

@pakmarkthub
Copy link
Collaborator

pakmarkthub commented May 30, 2025

@agirault The fix has been merged to R2.5 branch (https://github.com/NVIDIA/gdrcopy/tree/R2.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@drossetti drossetti drossetti left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@agirault @pakmarkthub @drossetti