so-rose
/
convmlv
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 3 Wiki Activity

Huge update: Multithreaded DNG, new argument parsing & long arguments, reuse root folders, 

global config file, more error checking, use s and e in frame range, DNG progress bar, bug fixes galore!!
modularize
Sofus Albert Høgsbro Rose 2016-05-18 17:36:28 -04:00
parent 9aeea608eb
commit dfa4d226cf
3 changed files with 594 additions and 406 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

172
README.md
View File

@ -11,134 +11,146 @@ Help page is below:
```text
Usage:
./convmlv.sh [OPTIONS] mlv_files
./convmlv.sh [FLAGS] [OPTIONS] files
INFO:
A script allowing you to convert .MLV, .RAW, or a folder with a DNG sequence into a sequence/movie with optional proxies. Images
are auto compressed. Many useful options are exposed, including formats (EXR by default).
A script allowing you to develop ML files into workable formats. Many useful options are exposed.
-->Image Defaults: Compressed 16-bit Linear EXR.
-->Acceptable Inputs: MLV, RAW, DNG Folder.
-->Option Input: From command line or config file.
VERSION: 1.8.2
VERSION: 1.9.0
DEPENDENCIES: If you don't use a feature, you don't need the dependency, though it's best to download them all.
-mlv_dump: For DNG extraction from MLV. http://www.magiclantern.fm/forum/index.php?topic=7122.0
MANUAL DEPENDENCIES:
-mlv_dump: Required. http://www.magiclantern.fm/forum/index.php?topic=7122.0
-raw2dng: For DNG extraction from RAW. http://www.magiclantern.fm/forum/index.php?topic=5404.0
-mlv2badpixels.sh: For bad pixel removal. https://bitbucket.org/daniel_fort/ml-focus-pixels/src
-dcraw: For RAW development.
-ffmpeg: For video creation.
-ImageMagick: Used for making proxy sequence.
-Python 3 + libs: Used for auto white balance.
-exiftool: Used in mlv2badpixels.sh.
-cr2hdr: For Dual ISO Development. Two links: http://www.magiclantern.fm/forum/index.php?topic=16799.0
-sRange.py: Required. See convmlv repository.
-balance.py: For Auto White Balance. See convmlv repository.
OPTIONS, BASIC:
-v version - Print out version string.
-o<path> OUTDIR - The path in which files will be placed (no space btwn -o and path).
-M<path> MLV_DUMP - The path to mlv_dump (no space btwn -M and path). Default is './mlv_dump'.
-R<path> RAW_DUMP - The path to raw2dng (no space btwn -M and path). Default is './raw2dng'.
-y<path> PYTHON - The path or command used to invoke Python. Defaults to python3.
-B<path> MLV_BP - The path to mlv2badpixels.sh (by dfort). Default is './mlv2badpixels.sh'.
-v, --version version - Print out version string.
-h, --help help - Print out this help page.
-T[int] Max process threads, for multithreaded parts of the program. Defaults to 8.
-C, --config CONFIG - Designates config file to use.
-o, --outdir <path> OUTDIR - The path in which files will be placed.
-P, --res-path <path> RES_PATH - The path in which all manual dependencies are looked for.
--mlv-dump <path> MLV_DUMP - The path to mlv_dump.
--raw-dump <path> RAW_DUMP - The path to raw2dng.
--badpixels <path> MLV_BP - The path to mlv2badpixels.sh (by dfort).
--cr-hdr <path> CR_HDR
--srange <path> SRANGE
--balance <path> BAL
--python <path> PYTHON - The path or command used to invoke Python.
-T, --threads [int] THREADS - Override amount of utilized process threads
OPTIONS, OUTPUT:
-i IMAGE - Specify to create an image sequence (EXR by default).
-i IMAGE - Will output image sequence.
-f[0:3] IMG_FMT - Create a sequence of <format> format. 0 is default.
-t [0:3] IMG_FMT - Specified image output format.
--> 0: EXR (default), 1: TIFF, 2: PNG, 3: Cineon (DPX)."
-c COMPRESS - Specify to turn ***off*** automatic image compression. Auto compression options otherwise used:
--> TIFF: ZIP (best for 16-bit), PIZ for EXR (best for grainy images), PNG: lvl 9 (zlib deflate), DPX: RLE.
-m MOVIE - Specify to create a Prores4444 video.
-m MOVIE - Will output a Prores4444 file.
-p[0:3] PROXY - Specifies the proxy mode. 0 is default.
--> 0: No proxies. 1: H.264 proxy. 2: JPG proxy sequence. 3: Both.
--> JPG proxy won't be developed w/o -i. H.264 proxy will be developed no matter what, if specified.
-p [0:3] PROXY - Create proxies alongside main output.
--> 0: No proxies (Default). 1: H.264 proxy. 2: JPG proxy sequence. 3: Both.
--> JPG proxy *won't* be developed w/o IMAGE. H.264 proxy *will* be developed no matter what, if specified.
-s[0%:100%] PROXY_SCALE - the size, in %, of the proxy output.
--> Use -s<percentage>% (no space). 50% is default.
-s [0%:100%] PROXY_SCALE - the size, in %, of the proxy output.
--> 50% is default.
-k KEEP_DNGS - Specify if you want to keep the DNG files.
--> If you run convmlv on the dng_<name> folder, you will reuse those DNGs - no need to redevelop!
-k KEEP_DNGS - Specify if you want to keep the DNG files.
--> Run convmlv on the top level folder of former output to reuse saved DNGs from that run!
-E<range> FRAME_RANGE - Specify to process only this frame range.
--> Use s and e appropriately to specify start and end.
--> <range> must be written as <start>-<end>, indexed from 0 to (# of frames - 1).
--> If you write a single number, only that frame will be developed.
-r <start>-<end> FRAME_RANGE - Specify to output this frame range only.
--> You may use s and e, such that s = start frame, e = end frame.
--> Indexed from 0 to (# of frames - 1).
--> A single number may be writted to develop that frame only.
--uncompress UNCOMP - Turns off lossless image compression. Otherwise:
--> TIFF: ZIP, EXR: PIZ, PNG: lvl 9 (zlib deflate), DPX: RLE.
OPTIONS, RAW DEVELOPMENT:
-d[0:3] DEMO_MODE - DCraw demosaicing mode. Higher modes are slower. 1 is default.
--> Use -d<mode> (no space). 0: Bilinear. 1: VNG (default). 2: PPG. 3: AHD.
-d [0:3] DEMO_MODE - Demosaicing algorithm. Higher modes are slower + better.
--> 0: Bilinear. 1: VNG (default). 2: PPG. 3: AHD.
-r FOUR_COLOR - Interpolate as four colors. Can often fix weirdness with VNG/AHD.
-f FOUR_COLOR - Interpolate as RGBG. Can often fix weirdness with VNG/AHD.
-H[0:9] HIGHLIGHT_MODE - 2 looks the best, but can break. 0 is a safe bet.
--> Use -H<number> (no space). 0 clips. 1 allows colored highlights. 2 adjusts highlights to grey.
--> 3 through 9 do highlight reconstruction with a certain tone. See dcraw documentation.
-H [0:9] HIGHLIGHT_MODE - Highlight management options.
--> 0: White, clipped highlights. 1: Clipped, colored highlights. 2: Similar to 1, but adjusted to grey.
--> 3-9: Highlight reconstruction. Can cause flickering; 1 or 2 usually give better results.
-C[0:3] CHROMA_SMOOTH - Apply chroma smoothing to the footage, which may help ex. with noise/bad pixels.
-c [0:3] CHROMA_SMOOTH - Apply shadow/highlight chroma smoothing to the footage.
--> 0: None (default). 1: 2x2. 2: 3x3. 3: 5x5.
--> Only applied to .MLV files.
--> MLV Only.
-n[int] NOISE_REDUC - This is the threshold of wavelet denoising - specify to use.
--> Use -n<number>. Defaults to no denoising. 150 tends to be a good setting; 350 starts to look strange.
-n [int] NOISE_REDUC - Apply wavelet denoising.
--> Default: None. Subtle: 50. Medium: 100. Strong: 200.
-g[0:4] SPACE - This is output color space. 0 is default.
--> Use -g<mode> (no space). 0: Linear. 1: 2.2 (Adobe RGB). 2: 1.8 (ProPhoto RGB). 3: sRGB. 4: BT.709.
-g [0:4] SPACE - Output color transformation.
--> 0: Linear. 1: 2.2 (Adobe RGB). 2: 1.8 (ProPhoto RGB). 3: sRGB. 4: BT.709.
-S SHALLOW - Specifying this option will create an 8-bit output instead of a 16-bit output.
--> It'll kind of ruin the point of RAW, though....
--shallow SHALLOW - Output 8-bit files.
OPTIONS, COLOR:
-w[0:2] WHITE - This is a modal white balance setting. Defaults to 1.
--> Use -w<mode> (no space).
--> 0: Auto WB (Requires Python Deps). 1: Camera WB. 2: No Change.
-w [0:2] WHITE - This is a modal white balance setting.
--> 0: Auto WB. 1: Camera WB (default). 2: No Change.
-L WHITE_SCALE - Specify to allow channels to clip as a result of any white balance.
--> Information loss occurs in certain situations.
-l <path> LUT - Specify a LUT to apply.
--> Supports cube, 3dl, dat, m3d.
--> LUT cannot be applied to EXR sequences.
-t[int] SATPOINT - Specify the 14-bit saturation point of your camera.
-S [int] SATPOINT - Specify the 14-bit saturation point of your camera.
--> Lower if -H1 yields purple highlights. Must be correct for highlight reconstruction.
--> Determine using the max value of 'dcraw -D -j -4 -T'
-A[int] WHITE_SPD - This is the amount of samples from which AWB will be calculated.
-->About this many frames, averaged over the course of the sequence, will be used to do AWB.
--white-speed [int] WHITE_SPD - Samples used to calculate AWB
-l<path> LUT - This is a path to the 3D LUT. Specify the path to the LUT to use it.
--> Compatibility determined by ffmpeg (.cube is supported).
--> LUT cannot be applied to EXR sequences.
--> Path to LUT (no space between -l and path).
--allow-white-clip WHITE_SCALE - Let White Balance multipliers clip.
OPTIONS, FEATURES:
-u DUAL_ISO - Process file as dual ISO.
-u DUAL_ISO - Process as dual ISO.
-b BADPIXELS - Fix focus pixels issue using dfort's script.
--> His file can be found at https://bitbucket.org/daniel_fort/ml-focus-pixels/src.
-b BADPIXELS - Fix focus pixels issue using dfort's script.
-a<path> BADPIXEL_PATH - Use, appending to the generated one, your own .badpixels file.
--> Use -a<path> (no space). How to: http://www.dl-c.com/board/viewtopic.php?f=4&t=686
-a <path> BADPIXEL_PATH - Use your own .badpixels file.
--> How to: http://www.dl-c.com/board/viewtopic.php?f=4&t=686
-F<path> DARKFRAME - This is the path to the dark frame MLV, for noise reduction.
--> This is a noise reduction technique: Record 5 sec w/lens cap on & same settings as footage.
--> Pass in that MLV file (not .RAW) as <path> to get noise reduction on all passed MLV files.
-F <path> DARKFRAME - This is the path to a "dark frame MLV"; effective for noise reduction.
--> How to: Record 5 sec w/lens cap on & same settings as footage. Pass MLV in here.
--> If the file extension is '.darkframe', the file will be used as the preaveraged dark frame.
-R <path> dark_out - Specify to create a .darkframe file from passed in MLV.
--> Outputs <arg>.darkframe file to <path>.
OPTIONS, INFO:
-e Output MLV settings.
-K Debian Package Deps - Lists dependecies. Works with apt-get on Debian; should be similar elsewhere.
--> No operations will be done.
--> Example: sudo apt-get install $ (./convmlv -K)
-e Output MLV settings.
-Y Python Deps - Lists Python dependencies. Works with pip.
--> No operations will be done.
--> Example: sudo pip3 install $ (./convmlv -Y)
-K Debian Package Deps - Output package dependecies.
--> Install (Debian only): sudo apt-get install $ (./convmlv -K)
-N Manual Deps - Lists manual dependencies, which must be downloaded by hand.
--> There's no automatic way to install these. See the forum post.
-Y Python Deps - Lists Python dependencies. Works directly with pip.
-->Install (Linux): sudo pip3 install $ (./convmlv -Y)
-N Manual Deps - Lists manual dependencies, which must be downloaded by hand.
--> There's no automatic way to install these. See http://www.magiclantern.fm/forum/index.php?topic=16799.0 .
CONFIG FILE:
Next to each option is an uppercased item, ex. OUTDIR. In a convmlv config file, you can specify this option
in the following format, line by line:
<VARNAME> <VALUE>
Some more notes regarding config files:
-The global config file, read every time, will be looked for in /home/sofus/convmlv.conf.
-Hashtags are considered comments, if and only if they are the first character in the line.
```

786
convmlv.sh
View File

File diff suppressed because it is too large Load Diff

42
sRange.py 100644
View File

@ -0,0 +1,42 @@
#!/usr/bin/env python3
#~ The MIT License (MIT)
#~ Copyright (c) 2016 Sofus Rose
#~ Permission is hereby granted, free of charge, to any person obtaining a copy
#~ of this software and associated documentation files (the "Software"), to deal
#~ in the Software without restriction, including without limitation the rights
#~ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
#~ copies of the Software, and to permit persons to whom the Software is
#~ furnished to do so, subject to the following conditions:
#~
#~ The above copyright notice and this permission notice shall be included in all
#~ copies or substantial portions of the Software.
#~
#~ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
#~ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
#~ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
#~ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
#~ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
#~ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
#~ SOFTWARE.
import sys
def spSeq(seq, outLen) :
perfSep = (1/outLen) * len(seq)
return list(filter(len, [seq[round(perfSep * i):round(perfSep * (i + 1))] for i in range(len(seq))]))
def splitThreadRange(inNum, inThreads) :
return [str(l[0]) + '-' + str(l[-1]) for l in spSeq(list(range(inNum)), inThreads)]
if __name__ == "__main__" :
num = int(sys.argv[1])
threads = int(sys.argv[2])
print(*splitThreadRange(num, threads))
#mlv_dump cannot use 0-0.