so-rose
/
convmlv
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 3 Wiki Activity

Finished release + bugfixes. See 1.9.4 changelog.

modularize
Sofus Albert Høgsbro Rose 2016-06-30 23:01:35 +02:00
parent e0bce02bba
commit b4622fc2a2
6 changed files with 325 additions and 50 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
CHANGELOG
View File

@ -1,4 +1,14 @@
[code] [code]
[b]v1.9.4[/b]: [url=https://bitbucket.org/so-rose/convmlv/src]https://bitbucket.org/so-rose/convmlv/src[/url]
[size=8pt]*Added Unsharp filter. Useful for counteracting all 6 denoisers ;).[/size]
[size=8pt]*Bug Fixes/Small Improvements: File-specific blocks/multiple files fixed. Filter application has been sanitized.[/size]
[b]v1.9.3[/b]: [url=https://bitbucket.org/so-rose/convmlv/src]https://bitbucket.org/so-rose/convmlv/src[/url]
[size=8pt]*Deshake Support! Just specify -D, and auto-stabilization will be applied. You may wish to scale it later.[/size]
[size=8pt]*HQDN3D Denoising Support. A superior denoising method (used in Handbrake). See 'convmlv -h' for usage instructions..[/size]
[size=8pt]*removegrain Denoising Support. A more specialized denoising method, ripe for experimentation. See https://ffmpeg.org/ffmpeg-filters.html#removegrain .[/size]
[size=8pt]*Bug Fixes: Different formats should now look the same. They didn't before... I hate color spaces... Misc. Bug fixes too.[/size]
[b]v1.9.2[/b]: [url=https://bitbucket.org/so-rose/convmlv/src]https://bitbucket.org/so-rose/convmlv/src[/url] [b]v1.9.2[/b]: [url=https://bitbucket.org/so-rose/convmlv/src]https://bitbucket.org/so-rose/convmlv/src[/url]
[size=8pt]*Atadenoise Support! It's a great denoising method, used with "-N <A>-<B>" See the documentation for how to configure A and B values.[/size] [size=8pt]*Atadenoise Support! It's a great denoising method, used with "-N <A>-<B>" See the documentation for how to configure A and B values.[/size]
[size=8pt]*General FFMPEG Filter Support: The framework for any ffmpeg -vf filters exists now! lut3d and atadenoise are currently implemented.[/size] [size=8pt]*General FFMPEG Filter Support: The framework for any ffmpeg -vf filters exists now! lut3d and atadenoise are currently implemented.[/size]

12
README.md
View File

@ -21,7 +21,7 @@ INFO:
--> Forum Post: http://www.magiclantern.fm/forum/index.php?topic=16799. --> Forum Post: http://www.magiclantern.fm/forum/index.php?topic=16799.
VERSION: 1.9.3 VERSION: 1.9.4
MANUAL DEPENDENCIES: MANUAL DEPENDENCIES:
-mlv_dump: Required. http://www.magiclantern.fm/forum/index.php?topic=7122.0 -mlv_dump: Required. http://www.magiclantern.fm/forum/index.php?topic=7122.0
@ -125,6 +125,15 @@ OPTIONS, COLOR:
-w [0:2] WHITE - This is a modal white balance setting. -w [0:2] WHITE - This is a modal white balance setting.
--> 0: Auto WB. 1: Camera WB (default). 2: No Change. --> 0: Auto WB. 1: Camera WB (default). 2: No Change.
-A [i:i:i:i] SHARP - Lets you sharpen, or blur, your footage.
--> Size/Strength (S/T). S is the size of the sharpen/blur effect, T is the strength of the sharpen/blur effect.
--> Luma/Chroma (L/C). L is the detail, C is the color. Luma sharpening more effective.
--> Tip: Chroma blur can actually be a helpful noise reduction technique.
--> Option Value: <LS>:<LT>:<CS>:<CT>
--> LS and CS must be ODD, between 3 and 63. Negative LT/CT values blur, while positive ones sharpen.
--> Strong Sharp: 7:3:7:3 Strong Blur: 7,-3:7,-3
-l <path> LUT - Specify a LUT to apply. -l <path> LUT - Specify a LUT to apply.
--> Supports cube, 3dl, dat, m3d. --> Supports cube, 3dl, dat, m3d.
@ -212,4 +221,5 @@ CONFIG FILE:
Contact me with any feedback or questions at convmlv@sofusrose.com, or PM me (so-rose) on the ML forums! Contact me with any feedback or questions at convmlv@sofusrose.com, or PM me (so-rose) on the ML forums!
``` ```

41
configs/atadenoise-test.conf
View File

@ -4,7 +4,6 @@ CONFIG_NAME atadenoise-test
### <------------------ BASIC OPTIONS ------------------> ### <------------------ BASIC OPTIONS ------------------>
OUTDIR ./raw_conv OUTDIR ./raw_conv
#RES_PATH /home/sofus/subhome/src/convmlv
MLV_DUMP /home/sofus/subhome/src/convmlv/mlv_dump MLV_DUMP /home/sofus/subhome/src/convmlv/mlv_dump
RAW_DUMP /home/sofus/subhome/src/convmlv/raw2dng RAW_DUMP /home/sofus/subhome/src/convmlv/raw2dng
@ -17,35 +16,51 @@ PYTHON python3
#~ THREADS 1 #~ THREADS 1
### <------------------ OUTPUT OPTIONS ------------------> ### <------------------ OUTPUT OPTIONS ------------------>
#~ IMAGE IMAGE
IMG_FMT 0 IMG_FMT 0
MOVIE #~ MOVIE
PROXY 0 #~ PROXY 3
#~ PROXY_SCALE 50% PROXY_SCALE 50%
KEEP_DNGS KEEP_DNGS
FRAME_RANGE 300-350
#~ UNCOMP #~ UNCOMP
### <------------------ RAW OPTIONS ------------------> ### <------------------ RAW OPTIONS ------------------>
DEMO_MODE 2 DEMO_MODE 3
HIGHLIGHT_MODE 0 #~ CHROMA_SMOOTH 1
#~ TEMP_NOISE 0.15-0.04 HIGHLIGHT_MODE 7
#~ HQ_NOISE 7-7:5-5 #~ TEMP_NOISE 0.1-0.04
HQ_NOISE 3-2:3-4
REM_NOISE 1-1-1-1 REM_NOISE 1-1-1-1
SPACE 3 SPACE 0
### <------------------ COLOR OPTIONS ------------------> ### <------------------ COLOR OPTIONS ------------------>
WHITE 1 WHITE 1
SHARP 7:5:5:3
### <------------------ FEATURE OPTIONS ------------------> ### <------------------ FEATURE OPTIONS ------------------>
#~ DESHAKE #~ DESHAKE
#DARKFRAME ./df.darkframe DARKFRAME /home/sofus/subhome/src/convmlv/darkframes_7D/f2.8_iso2500_crop1080.darkframe
#~ LUT /home/sofus/subhome/src/convmlv/test_footage/girl_lut_test.cube BADPIXEL_PATH ./7D_badpixels.txt
/ test_girl_noise
LUT /home/sofus/subhome/src/convmlv/test_footage/girl_lut_test.cube
FRAME_RANGE 456
*
/ M03-1714
FRAME_RANGE 10-50
*
/ short
WHITE 0
*

3
configs/template.txt
View File

@ -14,7 +14,7 @@ SRANGE /home/sofus/subhome/src/convmlv/sRange.py
BAL /home/sofus/subhome/src/convmlv/balance.py BAL /home/sofus/subhome/src/convmlv/balance.py
PYTHON python3 PYTHON python3
THREADS 7 #THREADS 7
### <------------------ OUTPUT OPTIONS ------------------> ### <------------------ OUTPUT OPTIONS ------------------>
IMAGE IMAGE
@ -50,6 +50,7 @@ SPACE 3
WHITE 0 WHITE 0
SHARP 7:3:0:0
#LUT lut.cube #LUT lut.cube
#SATPOINT 15000 #SATPOINT 15000

90
convmlv.sh
View File

@ -2,10 +2,10 @@
#TODO: #TODO:
#~ Stats for .RAW files. #~ Stats for .RAW files.
#~ Integrate anti-vertical banding. May require being able to use multiple darkframe files. #~ Color Systems - CLog, etc. .
#~ Better Preview: #~ Better Preview:
#~ --> A different module (like -e) for live viewing of footage, under convmlv settings. Danne is working on this :). #~ --> A different module (like -q or -R) for live viewing of footage, under convmlv settings. Danne is working on this :).
#BUG: Relative OUTDIR makes baxpixel generation fail if ./mlv2badpixels.sh doesn't exist. Fixed on Linux only. #BUG: Relative OUTDIR makes baxpixel generation fail if ./mlv2badpixels.sh doesn't exist. Fixed on Linux only.
#CONCERN: Weirdness with color spaces in general. No impact to user; just some weirdly placed sRGB conversions if'ed by DPX. #CONCERN: Weirdness with color spaces in general. No impact to user; just some weirdly placed sRGB conversions if'ed by DPX.
@ -36,7 +36,7 @@
#~ SOFTWARE. #~ SOFTWARE.
#BASIC VARS #BASIC VARS
VERSION="1.9.3" #Version string. VERSION="1.9.4" #Version string.
INPUT_ARGS=$(echo "$@") #The original input argument string. INPUT_ARGS=$(echo "$@") #The original input argument string.
if [[ $OSTYPE == "linux-gnu" ]]; then if [[ $OSTYPE == "linux-gnu" ]]; then
@ -115,6 +115,8 @@ setDefaults() { #Set all the default variables. Run here, and also after each AR
BADPIXEL_PATH="" BADPIXEL_PATH=""
isBP=false isBP=false
DARKFRAME="" DARKFRAME=""
useDF=false
DARK_PROC=""
SETTINGS_OUTPUT=false SETTINGS_OUTPUT=false
MK_DARK=false MK_DARK=false
DARK_OUT="" DARK_OUT=""
@ -130,6 +132,7 @@ setDefaults() { #Set all the default variables. Run here, and also after each AR
#FFMPEG Filters #FFMPEG Filters
FFMPEG_FILTERS=false #Whether or not FFMPEG filters are going to be used. FFMPEG_FILTERS=false #Whether or not FFMPEG filters are going to be used.
FILTER_ARR=()
TEMP_NOISE="" #Temporal noise reduction. TEMP_NOISE="" #Temporal noise reduction.
tempDesc="" tempDesc=""
LUT="" #lut3d LUT application LUT="" #lut3d LUT application
@ -147,7 +150,7 @@ setDefaults() { #Set all the default variables. Run here, and also after each AR
setDefaults #Run now, but also later. setDefaults #Run now, but also later.
help() { help() {
less -R << EOF cat << EOF
Usage: Usage:
$(echo -e "\033[1m./convmlv.sh\033[0m [FLAGS] [OPTIONS] \033[2mfiles\033[0m") $(echo -e "\033[1m./convmlv.sh\033[0m [FLAGS] [OPTIONS] \033[2mfiles\033[0m")
@ -263,7 +266,7 @@ OPTIONS, COLOR:
-w [0:2] WHITE - This is a modal white balance setting. -w [0:2] WHITE - This is a modal white balance setting.
--> 0: Auto WB. 1: Camera WB (default). 2: No Change. --> 0: Auto WB. 1: Camera WB (default). 2: No Change.
-A [i:i:i:i] SHARP - Lets you sharpen, or blur, your footage. -A [i:i:i:i] SHARP - Lets you sharpen, or blur, your footage.
--> Size/Strength (S/T). S is the size of the sharpen/blur effect, T is the strength of the sharpen/blur effect. --> Size/Strength (S/T). S is the size of the sharpen/blur effect, T is the strength of the sharpen/blur effect.
--> Luma/Chroma (L/C). L is the detail, C is the color. Luma sharpening more effective. --> Luma/Chroma (L/C). L is the detail, C is the color. Luma sharpening more effective.
--> Tip: Chroma blur can actually be a helpful noise reduction technique. --> Tip: Chroma blur can actually be a helpful noise reduction technique.
@ -400,20 +403,24 @@ evalConf() {
if [[ -z $file ]]; then return; fi if [[ -z $file ]]; then return; fi
fBlock=false #Whether or not we are in a file-specific block. fBlock=false #Whether or not we are in a file-specific block.
fID="" #The name of the file-specific block we're in.
while IFS="" read -r line || [[ -n "$line" ]]; do while IFS="" read -r line || [[ -n "$line" ]]; do
line=$(echo "$line" | sed -e 's/^[ \t]*//') #Strip leading tabs/whitespaces. line=$(echo "$line" | sed -e 's/^[ \t]*//') #Strip leading tabs/whitespaces.
if [[ `echo "${line}" | cut -c1-1` == "#" ]]; then continue; fi #Ignore comments if [[ `echo "${line}" | cut -c1-1` == "#" ]]; then continue; fi #Ignore comments
if [[ `echo "${line}" | cut -c1-1` == "/" && `echo "${line}" | cut -d$' ' -f2` == $TRUNC_ARG ]]; then if [[ `echo "${line}" | cut -c1-1` == "/" ]]; then
if [[ $fBlock == true ]]; then echo "\n\e[0;31m\e[1mWARNING: Nested blocks!!!\e[0m"; fi if [[ $fBlock == true ]]; then echo -e "\n\e[0;31m\e[1mWARNING: Nested blocks!!!\e[0m"; fi
fBlock=true fBlock=true
fID=`echo "${line}" | cut -d$' ' -f2`
continue
fi #Enter a file-specific block with /, provided the argument name is correct. fi #Enter a file-specific block with /, provided the argument name is correct.
if [[ `echo "${line}" | cut -c1-1` == "*" ]]; then fBlock=false; fi #Leave a file-specific block. if [[ `echo "${line}" | cut -c1-1` == "*" ]]; then fBlock=false; fi #Leave a file-specific block.
if [[ ($argOnly == false && $fBlock == false) || ($argOnly == true && $fBlock == true) ]]; then #Conditions under which to write values. #~ echo $argOnly $fBlock $fID ${TRUNC_ARG%.*} `echo "${line}" | cut -d$' ' -f1`
if [[ ($argOnly == false && $fBlock == false) || ( ($argOnly == true && $fBlock == true) && $fID == ${TRUNC_ARG%.*} ) ]]; then #Conditions under which to write values.
case `echo "${line}" | cut -d$' ' -f1` in case `echo "${line}" | cut -d$' ' -f1` in
"CONFIG_NAME") CONFIG_NAME=`echo "${line}" | cut -d$' ' -f2` #Not doing anything with this right now. "CONFIG_NAME") CONFIG_NAME=`echo "${line}" | cut -d$' ' -f2` #Not doing anything with this right now.
;; ;;
@ -615,7 +622,7 @@ evalConf() {
;; ;;
"BADPIXEL_PATH") BADPIXEL_PATH=`echo "${line}" | cut -d$' ' -f2` "BADPIXEL_PATH") BADPIXEL_PATH=`echo "${line}" | cut -d$' ' -f2`
;; ;;
"DARKFRAME") DARKFRAME=`echo "${line}" | cut -d$' ' -f2` "DARKFRAME") DARKFRAME=`echo "${line}" | cut -d$' ' -f2`; useDF=true
;; ;;
esac esac
fi fi
@ -925,6 +932,7 @@ parseArgs() { #Amazing new argument parsing!!!
;; ;;
F) F)
DARKFRAME=${OPTARG} DARKFRAME=${OPTARG}
useDF=true
;; ;;
R) R)
MK_DARK=true MK_DARK=true
@ -1147,11 +1155,16 @@ for ARG in "${FILE_ARGS_ITER[@]}"; do #Go through FILE_ARGS_ITER array, copied f
#The Very Basics #The Very Basics
BASE="$(basename "$ARG")" BASE="$(basename "$ARG")"
EXT="${BASE##*.}" EXT="${BASE##*.}"
DIRNAME=$(dirname "$ARG")
TRUNC_ARG="${BASE%.*}" TRUNC_ARG="${BASE%.*}"
SCALE=`echo "($(echo "${PROXY_SCALE}" | sed 's/%//') / 100) * 2" | bc -l` #Get scale as factor for halved video, *2 for 50% SCALE=`echo "($(echo "${PROXY_SCALE}" | sed 's/%//') / 100) * 2" | bc -l` #Get scale as factor for halved video, *2 for 50%
setBL=true setBL=true
joinArgs() { local d=$1; shift; echo -n "$1"; shift; printf "%s" "${@/#/$d}"; } joinArgs() { local d=$1; shift; echo -n "$1"; shift; printf "%s" "${@/#/$d}"; }
#Evaluate convmlv.conf configuration file for file-specific blocks.
evalConf "$LCONFIG" true
#Check that things exist.
checkDeps
#Construct the FFMPEG filters. #Construct the FFMPEG filters.
if [[ $FFMPEG_FILTERS == true ]]; then if [[ $FFMPEG_FILTERS == true ]]; then
@ -1160,13 +1173,10 @@ for ARG in "${FILE_ARGS_ITER[@]}"; do #Go through FILE_ARGS_ITER array, copied f
V_FILTERS_PROX="-vf $(joinArgs , ${HQ_NOISE} ${TEMP_NOISE} ${REM_NOISE} ${DESHAKE} ${SHARP} ${LUT} ${FINAL_SCALE})" #Proxy filter set adds the scale component. V_FILTERS_PROX="-vf $(joinArgs , ${HQ_NOISE} ${TEMP_NOISE} ${REM_NOISE} ${DESHAKE} ${SHARP} ${LUT} ${FINAL_SCALE})" #Proxy filter set adds the scale component.
#Created formatted array of filters, FILTER_ARR. #Created formatted array of filters, FILTER_ARR.
compFilters=()
declare -a compFilters=("${hqDesc}" "${tempDesc}" "${remDesc}" "${deshakeDesc}" "${sharpDesc}" "${lutDesc}") declare -a compFilters=("${hqDesc}" "${tempDesc}" "${remDesc}" "${deshakeDesc}" "${sharpDesc}" "${lutDesc}")
for v in "${compFilters[@]}"; do if test "$v"; then FILTER_ARR+=("$v"); fi; done for v in "${compFilters[@]}"; do if test "$v"; then FILTER_ARR+=("$v"); fi; done
fi fi
#Evaluate convmlv.conf configuration file for file-specific blocks.
evalConf "$LCONFIG" true
#Check that things exist.
checkDeps
#Potentially Print Settings #Potentially Print Settings
if [ $SETTINGS_OUTPUT == true ]; then if [ $SETTINGS_OUTPUT == true ]; then
@ -1174,7 +1184,7 @@ for ARG in "${FILE_ARGS_ITER[@]}"; do #Go through FILE_ARGS_ITER array, copied f
# Read the header for interesting settings :) . # Read the header for interesting settings :) .
mlvSet mlvSet
echo -e "\n\e[1m\e[0;32m\e[1mFile\e[0m\e[0m: ${ARG}\n" echo -e "\n\e[1m\e[0;32m\e[1mFile\e[0m: ${ARG}\n"
prntSet prntSet
continue continue
elif [ $EXT == "RAW" ] || [ $EXT == "raw" ]; then elif [ $EXT == "RAW" ] || [ $EXT == "raw" ]; then
@ -1208,10 +1218,18 @@ for ARG in "${FILE_ARGS_ITER[@]}"; do #Go through FILE_ARGS_ITER array, copied f
list="" list=""
for item in $remArr; do for item in $remArr; do
itemBase=$(basename $item)
itemExt="${itemBase##*.}"
itemDir=$(dirname "$item")
if [ -z "${list}" ]; then if [ -z "${list}" ]; then
list="${item}" if [[ $itemBase == $(basename $ARG) ]]; then
list="${itemDir}/\e[1m\e[32m${itemBase%.*}\e[0m.${itemExt}"
else
list="${itemDir}/\e[1m${itemBase%.*}\e[0m.${itemExt}"
fi
else else
list="${list}, ${item}" list="${list}, ${itemDir}/\e[1m${itemBase%.*}\e[0m.${itemExt}"
fi fi
done done
@ -1325,7 +1343,7 @@ for ARG in "${FILE_ARGS_ITER[@]}"; do #Go through FILE_ARGS_ITER array, copied f
fi fi
#Darkframe Averaging #Darkframe Averaging
if [ ! $DARKFRAME == "" ]; then if [[ $useDF == true ]]; then
echo -e "\e[1m${TRUNC_ARG}:\e[0m Creating darkframe for subtraction...\n" echo -e "\e[1m${TRUNC_ARG}:\e[0m Creating darkframe for subtraction...\n"
avgFrame="${TMP}/avg.darkframe" #The path to the averaged darkframe file. avgFrame="${TMP}/avg.darkframe" #The path to the averaged darkframe file.
@ -1382,7 +1400,7 @@ for ARG in "${FILE_ARGS_ITER[@]}"; do #Go through FILE_ARGS_ITER array, copied f
fileRanges=(`echo $($SRANGE $REAL_FRAMES $THREADS)`) #Get an array of frame ranges from the amount of frames and threads. I used a python script for this. fileRanges=(`echo $($SRANGE $REAL_FRAMES $THREADS)`) #Get an array of frame ranges from the amount of frames and threads. I used a python script for this.
#Looks like this: 0-1 2-2 3-4 5-5 6-7 8-8 9-10. Put that in an array. #Looks like this: 0-1 2-2 3-4 5-5 6-7 8-8 9-10. Put that in an array.
devDNG() { #Takes n arguments: 1{}, the frame range 2$MLV_DUMP 3$REAL_MLV 4$DARK_PROC 5$tmpOut 6$smooth 7$TMP 8$FRAME_END 9$TRUNC_ARG 10$FRAME_START devDNG() { #Takes n arguments: 1{}, the frame range 2$MLV_DUMP 3$REAL_MLV 4$DARK_PROC 5$no_data 6$smooth 7$TMP 8$FRAME_END 9$TRUNC_ARG 10$FRAME_START
range=$1 range=$1
firstFrame=false firstFrame=false
if [[ $range == "0-0" ]]; then #mlv_dump can't handle 0-0, so we develop 0-1. if [[ $range == "0-0" ]]; then #mlv_dump can't handle 0-0, so we develop 0-1.
@ -1420,7 +1438,7 @@ for ARG in "${FILE_ARGS_ITER[@]}"; do #Go through FILE_ARGS_ITER array, copied f
for range in "${fileRanges[@]}"; do echo $range; done | #For each frame range, assign a thread. for range in "${fileRanges[@]}"; do echo $range; done | #For each frame range, assign a thread.
xargs -I {} -P $THREADS -n 1 \ xargs -I {} -P $THREADS -n 1 \
bash -c "devDNG '{}' '$MLV_DUMP' '$REAL_MLV' '$DARK_PROC' '$tmpOut' '$smooth' '$TMP' '$FRAME_END' '$TRUNC_ARG' '$FRAME_START'" bash -c "devDNG '{}' '$MLV_DUMP' '$REAL_MLV' '$DARK_PROC' 'no_data' '$smooth' '$TMP' '$FRAME_END' '$TRUNC_ARG' '$FRAME_START'"
#Since devDNG must run in a subshell, globals don't follow. Must pass *everything* in. #Since devDNG must run in a subshell, globals don't follow. Must pass *everything* in.
echo -e "\e[2K\rMLV to DNG: Frame ${FRAME_END}/${FRAME_END}\c" #Ensure it looks right at the end. echo -e "\e[2K\rMLV to DNG: Frame ${FRAME_END}/${FRAME_END}\c" #Ensure it looks right at the end.
@ -1460,7 +1478,6 @@ for ARG in "${FILE_ARGS_ITER[@]}"; do #Go through FILE_ARGS_ITER array, copied f
bad_name="badpixels_${TRUNC_ARG}.txt" bad_name="badpixels_${TRUNC_ARG}.txt"
gen_bad="${TMP}/${bad_name}" gen_bad="${TMP}/${bad_name}"
touch $bad_name touch $bad_name
#~ exit
if [ $EXT == "MLV" ] || [ $EXT == "mlv" ]; then if [ $EXT == "MLV" ] || [ $EXT == "mlv" ]; then
$MLV_BP -o $gen_bad $ARG $MLV_BP -o $gen_bad $ARG
@ -1556,7 +1573,7 @@ for ARG in "${FILE_ARGS_ITER[@]}"; do #Go through FILE_ARGS_ITER array, copied f
echo -e "\e[1m${TRUNC_ARG}:\e[0m Generating WB...\n" echo -e "\e[1m${TRUNC_ARG}:\e[0m Generating WB...\n"
#Calculate n, the distance between samples. #Calculate n, the distance between samples.
frameLen=$(echo "$FRAME_END - $FRAME_START" | bc) frameLen=$(echo "$FRAME_END - $FRAME_START + 1" | bc) #Offset by one to avoid division by 0 errors later. min value must be 1.
if [[ $WHITE_SPD -gt $frameLen ]]; then if [[ $WHITE_SPD -gt $frameLen ]]; then
WHITE_SPD=$frameLen WHITE_SPD=$frameLen
fi fi
@ -1746,28 +1763,38 @@ for ARG in "${FILE_ARGS_ITER[@]}"; do #Go through FILE_ARGS_ITER array, copied f
echo -e "\n" echo -e "\n"
#FFMPEG Filter Application: Temporal Denoising, 3D LUTs, Deshake, hqdn Denoising, removegrain denoising so far. See construction of $V_FILTERS in PREPARATION. #FFMPEG Filter Application: Temporal Denoising, 3D LUTs, Deshake, hqdn Denoising, removegrain denoising, unsharp so far.
#See construction of $V_FILTERS in PREPARATION.
if [[ $FFMPEG_FILTERS == true ]]; then if [[ $FFMPEG_FILTERS == true ]]; then
tmpFiltered=${TMP}/filtered tmpFiltered="${TMP}/filtered"
tmpUnfiltered="${TMP}/unfiltered"
mkdir $tmpFiltered mkdir $tmpFiltered
mkdir $tmpUnfiltered
#Give correct output. #Give correct output.
echo -e "\e[1mApplying Filters:\e[0m $(joinArgs ", " "${FILTER_ARR[@]}")...\n" echo -e "\e[1mApplying Filters:\e[0m $(joinArgs ", " "${FILTER_ARR[@]}")...\n"
if [ $IMG_FMT == "exr" ]; then if [ $IMG_FMT == "exr" ]; then
echo -e "Note: EXR filtering lags before and after processing.\n" echo -e "Note: EXR filtering lags due to middle-step conversion (ffmpeg has no EXR encoder).\n"
img_res=$(identify ${SEQ}/${TRUNC_ARG}_$(printf "%06d" $(echo "$FRAME_START" | bc)).${IMG_FMT} | cut -d$' ' -f3) img_res=$(identify ${SEQ}/${TRUNC_ARG}_$(printf "%06d" $(echo "$FRAME_START" | bc)).${IMG_FMT} | cut -d$' ' -f3)
convert "${SEQ}/${TRUNC_ARG}_%06d.exr[${FRAME_START}-${FRAME_END}]" -set colorspace RGB dpx:- | \ i=0
ffmpeg -f image2pipe -vcodec dpx -s "${img_res}" -r $FPS -stats -i pipe:0 \ #Ideally, this would be threaded.
$V_FILTERS \ for devd in $SEQ/*.exr; do
-vcodec dpx -n -r $FPS -f image2pipe pipe:1 | \ convert $devd -set colorspace RGB "${tmpUnfiltered}/$(printf "%06d" ${i}).dpx"
convert -depth 16 - -colorspace RGB -compress piz -set colorspace RGB "${tmpFiltered}/%06d.${IMG_FMT}" let i++
done
ffmpeg -f image2 -vcodec dpx -s "${img_res}" -r $FPS -loglevel panic -stats -i "${tmpUnfiltered}/%06d.dpx" $V_FILTERS -vcodec dpx \
-n -r $FPS -f image2pipe pipe:1 | \
convert -depth 16 - -colorspace RGB -compress piz -set colorspace RGB "${tmpFiltered}/%06d.exr"
#For some reason, this whole process sends EXR's into sRGB. That's why -colorspace RGB is specified. See Nasty Hacks. #For some reason, this whole process sends EXR's into sRGB. That's why -colorspace RGB is specified. See Nasty Hacks.
else else
#Ideally, this would be all we need. But alas, ffmpeg + exr = breaks.
ffmpeg -start_number $FRAME_START -f image2 -i "${SEQ}/${TRUNC_ARG}_%06d.${IMG_FMT}" -loglevel panic -stats $V_FILTERS "${tmpFiltered}/%06d.${IMG_FMT}" ffmpeg -start_number $FRAME_START -f image2 -i "${SEQ}/${TRUNC_ARG}_%06d.${IMG_FMT}" -loglevel panic -stats $V_FILTERS "${tmpFiltered}/%06d.${IMG_FMT}"
fi fi
echo "" echo ""
#Replace the images in $SEQ with the filtered ones. #Replace the images in $SEQ with the filtered ones.
@ -1847,19 +1874,20 @@ for ARG in "${FILE_ARGS_ITER[@]}"; do #Go through FILE_ARGS_ITER array, copied f
rm -rf $TMP rm -rf $TMP
#MANUAL SANDBOXING - see note at the header of the loop. #MANUAL SANDBOXING - see note at the header of the loop.
set -- $INPUT_ARGS #Reset the argument input for reparsing again.
setDefaults #Hard reset everything. setDefaults #Hard reset everything.
OPTIND=1
evalConf "$GCONFIG" false #Rearse global config file. evalConf "$GCONFIG" false #Rearse global config file.
parseArgs "$@" #First, parse args all to set LCONFIG. parseArgs "$@" #First, parse args all to set LCONFIG.
shift $((OPTIND-1)) shift $((OPTIND-1))
OPTIND=1
evalConf "$LCONFIG" false #Parse local config file. evalConf "$LCONFIG" false #Parse local config file.
set -- $INPUT_ARGS #Reset the argument input for reparsing again, over the local config file. set -- $INPUT_ARGS #Reset the argument input for reparsing again, over the local config file.
OPTIND=1
parseArgs "$@" parseArgs "$@"
shift $((OPTIND-1)) shift $((OPTIND-1))
OPTIND=1
let ARGNUM-- let ARGNUM--
done done

211
docs/MANPAGE 100644
View File

@ -0,0 +1,211 @@
Usage:
./convmlv.sh [FLAGS] [OPTIONS] files
INFO:
A script allowing you to develop ML files into workable formats. Many useful options are exposed.
--> Image Defaults: Compressed 16-bit Linear EXR.
--> Acceptable Inputs: MLV, RAW, DNG Folder.
--> Option Input: From command line or config file.
--> Forum Post: http://www.magiclantern.fm/forum/index.php?topic=16799.
VERSION: 1.9.4
MANUAL DEPENDENCIES:
-mlv_dump: Required. http://www.magiclantern.fm/forum/index.php?topic=7122.0
-raw2dng: For DNG extraction from RAW. http://www.magiclantern.fm/forum/index.php?topic=5404.0
-mlv2badpixels.sh: For bad pixel removal. https://bitbucket.org/daniel_fort/ml-focus-pixels/src
-cr2hdr: For Dual ISO Development. Two links: http://www.magiclantern.fm/forum/index.php?topic=16799.0
-sRange.py: Required. See convmlv repository.
-balance.py: For Auto White Balance. See convmlv repository.
OPTIONS, BASIC:
-v, --version version - Print out version string.
-h, --help help - Print out this help page.
-C, --config config - Designates config file to use.
-o, --outdir <path> OUTDIR - The path in which files will be placed.
-P, --res-path <path> RES_PATH - The path in which all manual dependencies are looked for.
--mlv-dump <path> MLV_DUMP - The path to mlv_dump.
--raw-dump <path> RAW_DUMP - The path to raw2dng.
--badpixels <path> MLV_BP - The path to mlv2badpixels.sh (by dfort).
--cr-hdr <path> CR_HDR - The path to cr2hdr.
--srange <path> SRANGE - The path to sRange.py.
--balance <path> BAL - The path to balance.py.
--python <path> PYTHON - The path or command used to invoke Python.
-T, --threads [int] THREADS - Override amount of utilized process threads
OPTIONS, OUTPUT:
-i IMAGE - Will output image sequence.
-t [0:3] IMG_FMT - Specified image output format.
--> 0: EXR (default), 1: TIFF, 2: PNG, 3: Cineon (DPX)."
--> Note: Only EXR supports Linear output. Specify -g 3 if not using EXR.
-m MOVIE - Will output a Prores4444 file.
-p [0:3] PROXY - Create proxies alongside main output.
--> 0: No proxies (Default). 1: H.264 proxy. 2: JPG proxy sequence. 3: Both.
--> JPG proxy *won't* be developed w/o IMAGE. H.264 proxy *will* be developed no matter what, if specified.
-s [0%:100%] PROXY_SCALE - the size, in %, of the proxy output.
--> 50% is default.
-k KEEP_DNGS - Specify if you want to keep the DNG files.
--> Run convmlv on the top level folder of former output to reuse saved DNGs from that run!
-r <start>-<end> FRAME_RANGE - Specify to output this frame range only.
--> You may use s and e, such that s = start frame, e = end frame.
--> Indexed from 0 to (# of frames - 1).
--> A single number may be writted to develop that frame only.
--uncompress UNCOMP - Turns off lossless image compression. Otherwise:
--> TIFF: ZIP, EXR: PIZ, PNG: lvl 9 (zlib deflate), DPX: RLE.
OPTIONS, RAW DEVELOPMENT:
-d [0:3] DEMO_MODE - Demosaicing algorithm. Higher modes are slower + better.
--> 0: Bilinear. 1: VNG (default). 2: PPG. 3: AHD.
-f FOUR_COLOR - Interpolate as RGBG. Can often fix weirdness with VNG/AHD.
-H [0:9] HIGHLIGHT_MODE - Highlight management options.
--> 0: White, clipped highlights. 1: Clipped, colored highlights. 2: Similar to 1, but adjusted to grey.
--> 3-9: Highlight reconstruction. Can cause flickering; 1 or 2 usually give better results.
-c [0:3] CHROMA_SMOOTH - Apply shadow/highlight chroma smoothing to the footage.
--> 0: None (default). 1: 2x2. 2: 3x3. 3: 5x5.
--> MLV Only.
-n [int] WAVE_NOISE - Apply wavelet denoising.
--> Default: None. Subtle: 25. Medium: 50. Strong: 125.
-N <A>-<B> TEMP_NOISE - Apply temporal denoising.
--> A: 0 to 0.3. B: 0 to 5. A reacts to abrupt noise (splotches), B reacts to noise over time (fast motion causes artifacts).
--> Subtle: 0.03-0.04. High: 0.15-0.04. High, Predictable Motion: 0.15-0.07
-Q [i-i:i-i] HQ_NOISE - Apply 3D denoising filter.
--> In depth explanation: https://mattgadient.com/2013/06/29/in-depth-look-at-de-noising-in-handbrake-with-imagevideo-examples/ .
--> Spacial/Temporal (S/T). S will soften/blur/smooth, T will remove noise without doing that but may create artifacts.
--> Luma/Chroma (L/C). L is the detail, C is the color. Each one's denoising may be manipulated Spacially or Temporally.
--> Option Value: <LS>-<CS>:<LT>-<CT>
--> Weak: 2-1:2-3. Medium: 3-2:2-3. Strong: 7-7:5-5
-G [i-i-i-i] REM_NOISE - Yet another spatial denoiser, with 4 choices of 24 modes.
--> See https://ffmpeg.org/ffmpeg-filters.html#removegrain for list of modes.
--> Option Value: <mode1>-<mode2>-<mode3>-<mode4>
--> I truly cannot tell you what values will be helpful to you; there are too many... Look at the link above!
-g [0:4] SPACE - Output color transformation.
--> 0: Linear. 1: 2.2 (Adobe RGB). 2: 1.8 (ProPhoto RGB). 3: sRGB. 4: BT.709.
--shallow SHALLOW - Output 8-bit files.
OPTIONS, COLOR:
-w [0:2] WHITE - This is a modal white balance setting.
--> 0: Auto WB. 1: Camera WB (default). 2: No Change.
-A [i:i:i:i] SHARP - Lets you sharpen, or blur, your footage.
--> Size/Strength (S/T). S is the size of the sharpen/blur effect, T is the strength of the sharpen/blur effect.
--> Luma/Chroma (L/C). L is the detail, C is the color. Luma sharpening more effective.
--> Tip: Chroma blur can actually be a helpful noise reduction technique.
--> Option Value: <LS>:<LT>:<CS>:<CT>
--> LS and CS must be ODD, between 3 and 63. Negative LT/CT values blur, while positive ones sharpen.
--> Strong Sharp: 7:3:7:3 Strong Blur: 7,-3:7,-3
-l <path> LUT - Specify a LUT to apply.
--> Supports cube, 3dl, dat, m3d.
-S [int] SATPOINT - Specify the 14-bit saturation point of your camera. You don't usually want to.
--> Lower if -H1 yields purple highlights. Must be correct for highlight reconstruction.
--> Determine using the max value of 'dcraw -D -j -4 -T'
--white-speed [int] WHITE_SPD - Samples used to calculate AWB
--allow-white-clip WHITE_CLIP - Let White Balance multipliers clip.
OPTIONS, FEATURES:
-D DESHAKE - Stabilize the video using the wonderful ffmpeg "deshake" module.
--> You'll probably wish to crop/scale the output in editing, to avoid edge artifacts.
-u DUAL_ISO - Process as dual ISO.
-b BADPIXELS - Fix focus pixels issue using dfort's script.
-a <path> BADPIXEL_PATH - Use your own .badpixels file.
--> How to: http://www.dl-c.com/board/viewtopic.php?f=4&t=686
-F <path> DARKFRAME - This is the path to a "dark frame MLV"; effective for noise reduction.
--> How to: Record 5 sec w/lens cap on & same settings as footage. Pass MLV in here.
--> If the file extension is '.darkframe', the file will be used as the preaveraged dark frame.
-R <path> dark_out - Specify to create a .darkframe file from passed in MLV.
--> Usage: 'convmlv -R <path> <input>.MLV'
--> Averages <input>.MLV to create <path>.darkframe.
--> THE .darkframe EXTENSION IS ADDED FOR YOU.
OPTIONS, INFO:
-q Output MLV settings.
-K Debian Package Deps - Output package dependecies.
--> Install (Debian only): sudo apt-get install $ (./convmlv -K)
-Y Python Deps - Lists Python dependencies. Works directly with pip.
-->Install (Linux): sudo pip3 install $ (./convmlv -Y)
-M Manual Deps - Lists manual dependencies, which must be downloaded by hand.
--> There's no automatic way to install these. See http://www.magiclantern.fm/forum/index.php?topic=16799.0 .
CONFIG FILE:
Config files, another way to specify options, can save you time & lend you convenience in production situations.
GLOBAL: /home/sofus/convmlv.conf
LOCAL: Specify -C/--config.
SYNTAX:
Most options listed above have an uppercased VARNAME, ex. OUTDIR. Yu can specify such options in config files, as such:
<VARNAME> <VALUE>
One option per line only. Indentation by tabs or spaces is allowed, but not enforced.
Comments Lines starting with # are comments.
You may name a config using:
CONFIG_NAME <name>
Flags If the value is a true/false flag (ex. IMAGE), simply specifying VARNAME is enough. THere is no VALUE.
OPTION ORDER OF PRECEDENCE Options override each other as such:
-LOCAL options overwrite GLOBAL options.
-COMMAND LINE options overwrite LOCAL & GLOBAL options.
-FILE SPECIFIC options overwrite ALL ABOVE options.
File-Specific Block: A LOCAL config file lets you specify options for specific input names:
/ <TRUNCATED INPUTNAME>
...options here will only be
*
You must use the truncated (no .mlv or .raw) input name after the /. Nested blocks will fail.
With a single config file, you can control the development options of multiple inputs as specifically and/or generically
as you want. Batch developing everything can then be done with a single, powerful commmand.
Contact me with any feedback or questions at convmlv@sofusrose.com, or PM me (so-rose) on the ML forums!