Millions of previews

[SimplyCorbett]

I need to generate a couple million preview images.

Is there a way to speed this up other then what's listed in #107 ?

Is there a way to run multiple previewgenerator processes at the same time in parallel?

I have the optimizations of 60 jpeg quality set, etc. But it's still a very slow process - and infact, it generates faster by just browsing the gallery on nextcloud.

Interestingly, viewing the files via file manager with webdav is significantly faster at (local) preview generation.

[wuast94]

multithreated genereation would be nice, have 24C/48T and its using only one threat

[st3iny]

The PR #385 could help with this.

[octylFractal]

I used the following commands to do parallel generation, spawning one process, up to 2x CPU count, for each leaf directory that isn't marked with .nomedia:

{ find "$NEXTCLOUD_ROOT/$USER_ID/files/" -type d \( -exec [ -f '{}/.nomedia' ] \; -prune -o -print \); echo; } | awk 'index($0,prev"/")!=1 && NR!=1 {print prev} 1 {sub(/\/$/,""); prev=$0}' | sed "s|$NEXTCLOUD_ROOT||" | parallel --line-buffered --jobs '200%' occ preview:generate-all -vvv --path='{}'

Replace or set $NEXTCLOUD_ROOT with the root data directory, $USER_ID with the user to generate for, and occ with however you run occ.

This isn't very efficient due to the overhead of starting each process, but it's at least something.

[st3iny]

This is now possible.

Ref

• feat: implement spawning multiple workers for generate-all #577
• feat: allow running multiple instances of pre-generate #576