so-rose
/
convmlv
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 3 Wiki Activity

Using array; start wipe of globals. Need to complete arg functions, globals transition, and modularization of develop() before a lot of code cleanup, then the plugin system. That'll deserve a merge, where I'll branch into the big califrames, then start committing modular features. Before 2.1.0 I'll do some RCs for myself, reevaluate the colorspace bug, and that'll be it!

modularize
Sofus Albert Høgsbro Rose 2016-12-08 22:37:22 -05:00
parent 054d282ba6
commit 6390f75ff1
Signed by: so-rose
GPG Key ID: 3D01BE95F3EFFEB9
14 changed files with 697 additions and 609 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
DEPENDENCIES
View File

@ -1,22 +0,0 @@
Besides the tarball, you need a few external deps.
For your distribution:
First, install libc6:i386 for 64-bit systems.
Debian: sudo apt-get install $(./convmlv.sh -K 0)
Ubuntu: sudo apt-get install $(./convmlv.sh -K 1)
Fedora: sudo yum install $(./convmlv.sh -K 2)
Mac (Homebrew): brew install $(./convmlv.sh -K 3)
For Python (make sure you're using python3):
sudo python3 -m pip install $(./convmlv -Y)
Binary Source Code:
mlv2badpixels.sh: http://www.magiclantern.fm/forum/index.php?topic=16054
raw2dng: http://www.magiclantern.fm/forum/index.php?topic=5404.0
mlv_dump: http://www.magiclantern.fm/forum/index.php?topic=7122.0
cr2hdr: http://www.magiclantern.fm/forum/index.php?topic=7139.0

309
README.md
View File

@ -3,305 +3,42 @@
RAW ML Formats to Workable Images/Movies Developer RAW ML Formats to Workable Images/Movies Developer
======= =======
I needed a workflow to provide me full control over development for filmmaking with Magic Lantern (beyond mlrawviewer's excellent preview!), while keeping things simple by providing defaults for everything. I needed a workflow to provide me full control over development for filmmaking with Magic Lantern (beyond mlrawviewer's excellent preview!), while keeping things simple by providing defaults for everything.
So, I wrote this: A converter from ML formats (RAW, MLV, DNG sequences) to many usable formats (EXR, DPX, and yes, Prores) with many, many features in between!
Generically, as a DNG developer, it's very capable as well, even if all the ML formats are ignored. **So**, I wrote this: A program that converts from RAW formats (MLV, RAW, DNG, CR2) to workable formats (EXR, MOV, DPX, DNxHD, MJPEG, etc.) with many, many features in between (color management, filters, dual iso, demosaic, etc.)
I make shorts with convmlv :D so here's some examples of it (back at v1.7) in action!! It's extensible, letting you write plugins controlling/filtering any leg of the journey from raw to developed. Config file syntax also makes it a kind of "make for productions"; with configs and one command, you can develop your whole project from unsorted raw footage to editable, proxied, organized stuff!
I use it myself :D so here's some shorts with it (back at v1.7) in action!!
http://youtu.be/yi-G7sXHB1M http://youtu.be/yi-G7sXHB1M
http://youtu.be/yi-G7sXHB1M http://youtu.be/yi-G7sXHB1M
See [http://www.magiclantern.fm/forum/index.php?topic=16799.0](http://www.magiclantern.fm/forum/index.php?topic=16799.0) for more info. ## Installation
Supported platforms are Linux and Mac. To install, copy/paste the following into a terminal:
**The pdf found in the source and in the release, under docs->docs.pdf, is up to date. Refer to it for tutorials, tips, documentation, etc.!!** 1. **Extract tarball**. Rename it "convmlv" and put it somewhere useful. `cd` to that directory.
2. **Install Deps**: `sudo apt-get install $(./convmlv.sh -K <number>)`. Replace <number> as indicated below:
* For Debain, use the number "0"
* For Fedora, replace `apt-get` with `yum` and use the number "2".
* For Mac, replace `sudo apt-get` with `brew` and use the number "3". *Requires Homebrew to be installed.
Full help page is below: 3. **Install Python deps**: `sudo python3 -m pip install $ (./convmlv.sh -Y)`
4. (Optional) **Add to Path**: `sudo ln -s $(pwd)/convmlv.sh /usr/local/bin/convmlv`
```text You're done! For more info, see the Documentation (especially the PDF has a guide to this).
Usage:
./convmlv.sh [FLAGS] [OPTIONS] files
INFO: ## Documentation
A program allowing you to develop ML files into workable formats. Many useful options are exposed. Especially the PDF is designed as a tutorial; take a look!
--> Defaults: Compressed 16-bit Linear EXR. 10-bit Prores4444 MOV.
--> Color Defaults: Linear (1.0) Gamma on sRGB Gamut, using Camera White Balance.
--> Acceptable Inputs: MLV, RAW (requires raw2dng), Folder containing DNGs. **The Forum Thread** at [http://www.magiclantern.fm/forum/index.php?topic=16799.0](http://www.magiclantern.fm/forum/index.php?topic=16799.0) is always up to date.
--> Option Input: From command line or config file (specify with -C).
--> Forum Post: http://www.magiclantern.fm/forum/index.php?topic=16799. **The PDF** found *in the release* is up to date, including tutorials, tips, explanations, plugin API, and more.
--> A note: BE CAREFUL WITH OPTIONS. Wrong values will give very strange errors. Read this page well!!
It's as simple or complex as you need it to be: 'convmlv -m <mlvfile>.mlv' is enough for good-looking output! **The Help Page** is the most up to date; just run `convmlv -h` or look at docs-->MANPAGE in the release tarball.
VERSION: 2.0.1 ## Development
To build the PDF and MANPAGE using LATEX, run `docs/buildDocs.sh`. You can clean it up using `docs/cleanDocs.sh`.
MANUAL DEPENDENCIES: To make a release tarball, simply place all the manual binary deps into the `binaries` dir and run `./mkrelease.sh`.
Place these in RES_PATH (see OPTIONS, BASIC). Keep in mind you also need dist. and pip packages.
--> See 'Dist Deps' and 'Python Deps'
-- mlv_dump: Required binary. http://www.magiclantern.fm/forum/index.php?topic=7122.0 The source code is located in `src`, where you'll find bash and Python code, as well as the builtin plugins. convmlv.sh itself imports all the src's, and acts as a kind of program selector; develop is but one option.
-- color-core: Required folder of LUTs. See convmlv repository.
-- sRange.py: Required script. See convmlv repository.
-- raw2dng: For DNG extraction from RAW. http://www.magiclantern.fm/forum/index.php?topic=5404.0
-- mlv2badpixels.sh: For bad pixel removal. https://bitbucket.org/daniel_fort/ml-focus-pixels/src
-- cr2hdr: For Dual ISO Development. Two links: http://www.magiclantern.fm/forum/index.php?topic=16799.0
-- balance.py: For Auto White Balance. See convmlv repository.
-- color-ext: Extra LUTs, providing more color resources. See convmlv repository.
OPTIONS, BASIC:
-v, --version version - Print out version string.
-h, --help help - Print out this help page.
-C, --config config - Designates config file to use.
-o, --outdir <path> OUTDIR - The path in which files will be placed.
-P, --res-path <path> RES_PATH - The path in which all manual dependencies are looked for.
--> Default: Current Directory.
--dcraw <path> DCRAW - The path to dcraw.
--mlv-dump <path> MLV_DUMP - The path to mlv_dump.
--raw-dump <path> RAW_DUMP - The path to raw2dng.
--badpixels <path> MLV_BP - The path to mlv2badpixels.sh (by dfort).
--cr-hdr <path> CR_HDR - The path to cr2hdr.
--srange <path> SRANGE - The path to sRange.py.
--balance <path> BAL - The path to balance.py.
--python <path> PYTHON - The path or command used to invoke Python 3. Default is python3 on Linux, python on Mac.
-T, --threads [int] THREADS - Override amount of utilized process threads. Default is MAX - 1.
OPTIONS, OUTPUT:
-i IMAGE - Will output image sequence.
-t [0:3] IMG_FMT - Image output format.
--> 0: EXR (default), 1: TIFF, 2: PNG, 3: Cineon (DPX)."
-m MOVIE - Will output a Prores4444 file.
-p [0:3] PROXY - Create proxies alongside main output.
--> 0: No proxies (Default). 1: H.264 proxy. 2: JPG proxy sequence. 3: Both.
--> JPG proxy will always be in sRGB Gamma/sRGB Gamut. H.264 proxy is color managed.
--> JPG proxy *won't* be developed w/o IMAGE. H.264 proxy *will* be developed no matter what, if specified here.
--> Why? JPG is for potential use in editing. H.264 is for a quick visual preview of convmlv's output.
-s [0%:100%] PROXY_SCALE - the size, in %, of the proxy output.
--> Default: 50%.
-k KEEP_DNGS - Specify if you want to keep the DNG files.
--> Run convmlv on the top level folder of former output to reuse saved DNGs from that run!
-r <start>-<end> FRAME_RANGE - Specify to process an integer frame range.
--> You may use the characters 's' and 'e', such that s = start frame, e = end frame.
--> Indexed from 0 to (# of frames - 1). Develops from 1 to ($ of frames)
--> A single number may be writted to develop that single frame.
--> DO NOT try to reuse DNGs while developing a larger frame range.
--uncompress UNCOMP - Turns off lossless image compression. Otherwise:
--> TIFF: ZIP, EXR: PIZ, PNG: lvl 0, DPX: RLE.
OPTIONS, RAW DEVELOPMENT:
-d [0:3] DEMO_MODE - Demosaicing algorithm. Higher modes are slower + better.
--> 0: Bilinear. 1: VNG (default). 2: PPG. 3: AHD.
-f FOUR_COLOR - Interpolate as RGBG. Fixes weirdness with VNG/AHD, at the cost of sharpness.
-H [0:9] HIGHLIGHT_MODE - Highlight management options.
--> 0: White, clipped highlights. 1: Unclipped but colored highlights. 2: The defail of 1, but adjusted to grey.
--> 3-9: Highlight reconstruction. Can cause flickering. Start at 5, then adjust to color (down) or to white (up).
-c [0:3] CHROMA_SMOOTH - Apply shadow/highlight chroma smoothing to the footage.
--> 0: None (default). 1: 2x2. 2: 3x3. 3: 5x5.
--> MLV input Only.
-n [int] WAVE_NOISE - Apply wavelet denoising.
--> Default: None. Subtle: 25. Medium: 50. Strong: 125.
-N <A>-<B> TEMP_NOISE - Apply temporal denoising.
--> A: 0 to 0.3. B: 0 to 5. A reacts to abrupt noise (splotches), B reacts to noise over time (fast motion causes artifacts).
--> Subtle: 0.03-0.04. High: 0.15-0.04. High, Predictable Motion: 0.15-0.07
-Q [i-i:i-i] HQ_NOISE - Apply 3D denoising filter.
--> In depth explanation: https://mattgadient.com/2013/06/29/in-depth-look-at-de-noising-in-handbrake-with-imagevideo-examples/ .
--> Spacial/Temporal (S/T). S will soften/blur/smooth, T will remove noise without doing that but may create artifacts.
--> Luma/Chroma (L/C). L is the detail, C is the color. Each one's denoising may be manipulated Spacially or Temporally.
--> Option Value: <LS>-<CS>:<LT>-<CT>
--> Weak: 2-1:2-3. Medium: 3-2:2-3. Strong: 7-7:5-5
--> DONT combine with TEMP_NOISE.
-O [i-i-i-i] REM_NOISE - Yet another spatial denoiser, with 4 choices of 24 modes.
--> See https://ffmpeg.org/ffmpeg-filters.html#removegrain for list of modes.
--> Option Value: <mode1>-<mode2>-<mode3>-<mode4>
--> I truly cannot tell you what values will be helpful to you; there are too many... Look at the link!
--shallow SHALLOW - Output smaller, 8-bit files.
--> Read why this is a bad idea: http://www.cambridgeincolour.com/tutorials/bit-depth.htm
OPTIONS, COLOR:
-g [0:4] GAMMA - Output gamma. A curve applied to the output, for easier viewing/grading.
--> 0: Standard (Around 2.2). 1: Linear (Default).
--> Requires color-ext: 2: Cineon. 3: C-Log2 4: S-Log3
--> "Standard" grades to the gamut specification, and to 2.2 if that's not given.
-G [0:6] GAMUT - Output gamut. The range of colors that can exist in the output.
--> 0: sRGB (Default). 1: Adobe RGB. 2: Rec.709. 3: XYZ (Always Linear Gamma).
--> Requires color-ext: 4: Rec2020 5: DCI-P3 6: Sony S-Gamut3.cine
-w [0:2] WHITE - This is a modal white balance setting.
--> 0: Auto WB (requires balance.py). 1: Camera WB (default). 2: No Change.
--> AWB uses the Grey's World algorithm.
-A [i:i:i:i] SHARP - Lets you sharpen, or blur, your footage.
--> BE CAREFUL. Wrong values will give you strange errors.
--> Size/Strength (S/T). S is the size of the sharpen/blur effect, T is the strength of the sharpen/blur effect.
--> Luma/Chroma (L/C). L is the detail, C is the color. Luma sharpening more effective.
--> Option Value: <LS>:<LT>:<CS>:<CT>
--> LS and CS must be ODD, between 3 and 63. Negative LT/CT values blur, while positive ones sharpen.
--> Strong Sharp: 7:3:7:3 Strong Blur: 7,-3:7,-3. Medium Sharp: 5:1:3:0
-l <path> LUT - Specify a LUT to apply after Color Management.
--> Supports cube, 3dl, dat, m3d.
--> Specify -l multiple times, to apply multiple LUTs in sequence.
-S [int] SATPOINT - Specify the 14-bit uint saturation point of your camera. You don't usually need to.
--> Worth setting globally, as it's a per-camera setting. Must be correct for highlight reconstruction/unclipped highlights.
--> Lower from 15000 if -H1 yields purple highlights, until they turn white.
--> You can determine the optimal value using the max pixel value of 'dcraw -D -j -4 -T'.
--white-speed [int] WHITE_SPD - Manually specify samples used to calculate AWB.
--allow-white-clip WHITE_CLIP - Let the White Balance multipliers clip.
OPTIONS, FEATURES:
-D DESHAKE - Auto-stabilize the video using ffmpeg's "deshake" module.
--> You may wish to crop/scale the output later, to avoid edge artifacts.
-u DUAL_ISO - Process as dual ISO.
--> Requires cr2hdr.
-b BADPIXELS - Fix focus pixels issue using dfort's script.
--> Requires mlv2badpixels.sh.
-a <path> BADPIXEL_PATH - Use your own .badpixels file. Does NOT require mlv2badpixels.sh
--> How to: http://www.dl-c.com/board/viewtopic.php?f=4&t=686
-F <path> DARKFRAME - This is the path to a "dark frame MLV"; effective for noise reduction.
--> How to: Record 5 sec w/lens cap on & same settings as footage. Pass MLV in here.
--> If the file extension is '.darkframe', the file will be used as a preaveraged dark frame.
-R <path> darkframe_output - Specify to create a .darkframe file from passed in MLV.
--> Usage: 'convmlv -R <path> <input>.MLV'
--> Averages <input>.MLV to create <path>.darkframe.
--> THE .darkframe EXTENSION IS ADDED FOR YOU.
OPTIONS, INFO:
-q settings - Output MLV settings.
-K [0:3] Dist Deps - Output package dependecies, for use with common package managers.
--> 0: Debian, 1: Ubuntu, 2: Fedora, 3: Homebrew (Mac)
--> Deps Install (Debian): sudo apt-get install $(./convmlv.sh -K 0)
--> Deps Install (Ubuntu): sudo apt-get install $(./convmlv.sh -K 1)
--> Deps Install (Fedora): sudo yum install $(./convmlv.sh -K 2)
--> Deps Install (Homebrew Mac): brew install $(./convmlv.sh -K 3)
-Y Python Deps - Lists Python dependencies. Works directly with pip.
-->Install (Cross-Platform): sudo python3 -m pip install $ (./convmlv -Y)
-M Manual Deps - Lists manual dependencies, which must be downloaded by hand.
--> Manually place all in RES_PATH. See http://www.magiclantern.fm/forum/index.php?topic=16799.0 .
COLOR MANAGEMENT:
INTRO Images aren't simple. They are often stored, processed, and viewed as a result of complex transformations usually called,
in applications, color management. Understanding it is required as a colourist, and encouraged as a DPs and Cinematographers.
--> Intro: http://www.cambridgeincolour.com/tutorials/color-management1.htm
--> Understanding Gamma: http://www.cambridgeincolour.com/tutorials/gamma-correction.htm
--> Color Spaces: http://www.cambridgeincolour.com/tutorials/color-spaces.htm
--> Conversions: http://www.cambridgeincolour.com/tutorials/color-space-conversion.htm
--> Monitor Calibration: http://www.cambridgeincolour.com/tutorials/monitor-calibration.htm
PIPELINE convmlv is a color managed application, designed to retain quality from RAW footage:
-- mlv_dump writes camera-specific color transformation matrices as metadata in developed DNGs.
--> This defines the camera's gamut.
-- dcraw applies these matrices, then transforms the newly developed image to the super-wide XYZ colorspace.
--> All color detail is preserved, and if now in a well-defined colorspace.
--> No gamma has been applied - the image is now Linear XYZ.
-- ffmpeg applies the specified (up to) x64 resolution 3D LUTs, in .cube format.
--> Use -g and -G to specify Gamma/Gamut combinations.
--> The output employs DATA, not LEGAL, values. The resulting image is flatter, but retains all shadow/highlight detail.
--> -l specified LUTs are applied afterwards.
3D LUTS The included LUTs, found in the convmlv repository, are key to convmlv's color management solution:
-- color-core: The required LUTs, including sRGB (default), Adobe RGB, and Rec709 in Standard and Linear gamma.
-- color-ext: Optional LUTs, including Rec2020, DCI-P3, etc. in Standard/Linear gammas, but also in Log formats.
Create Your Own LUTs using LUTCalc, for any grading format output: https://cameramanben.github.io/LUTCalc/ (watch his tutorials).
--> Note that convmlv only accepts up to 64x64x64 LUTs. You can resize LUTs using pylut (https://pypi.python.org/pypi/pylut).
--> The pylut command to resize is 'pylut <yourx65lut>.cube --resize 64'. Alternatively, you can use pylut from Python (2X only).
--> I reccommend Legal --> Data LUTs, as this conserves shadow/highlight detail for grading. Legal --> Legal looks better, but with detail loss.
CONFIG FILE:
Config files, another way to specify options, can save you time & lend you convenience in production situations.
GLOBAL: /home/sofus/convmlv.conf
LOCAL: Specify -C/--config.
SYNTAX:
Most options listed above have an uppercased VARNAME, ex. OUTDIR. You can specify such options in config files, as such:
<VARNAME> <VALUE>
One option per line only. Indentation by tabs or spaces is allowed, but not enforced.
Comments Lines starting with # are comments.
You may name a config using:
CONFIG_NAME <name>
Flags If the value is a true/false flag (ex. IMAGE), simply specifying VARNAME is enough. There is no VALUE.
OPTION ORDER OF PRECEDENCE Options override each other as such:
-LOCAL options overwrite GLOBAL options.
-COMMAND LINE options overwrite LOCAL & GLOBAL options.
-FILE SPECIFIC options overwrite ALL ABOVE options.
File-Specific Block: A LOCAL config file lets you specify options for specific input names:
/ <TRUNCATED INPUTNAME>
...options here will only be
*
You must use the truncated (no .mlv or .raw) input name after the /. Nested blocks will fail.
With a single config file, you can control the development options of multiple inputs as specifically and/or generically
as you want. Batch developing everything can then be done with a single, powerful commmand.
Contact me with any feedback or questions at convmlv@sofusrose.com, PM me (so-rose) on the ML forums, or post on the thread!
```
Run 'sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" docs/MANPAGE' to strip colors from MANPAGE after generating through less. This is done automatically in 'docs/buildDocs.sh.

43
binaries/DEPENDENCIES 100644
View File

@ -0,0 +1,43 @@
Besides the tarball, you need a few external deps.
For your distribution:
First, install libc6:i386 for 64-bit systems.
Debian: sudo apt-get install $(./convmlv.sh -K 0)
Ubuntu: sudo apt-get install $(./convmlv.sh -K 1)
Fedora: sudo yum install $(./convmlv.sh -K 2)
Mac (Homebrew): brew install $(./convmlv.sh -K 3)
For Python (make sure you're using python3):
sudo python3 -m pip install $(./convmlv -Y)
Binary Source Code, distributed under GPLv2:
# Copyright (C) 2016 Magic Lantern Team
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the
# Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor,
# Boston, MA 02110-1301, USA.
#/
Source code found at links below:
mlv2badpixels.sh: http://www.magiclantern.fm/forum/index.php?topic=16054
raw2dng: http://www.magiclantern.fm/forum/index.php?topic=5404.0
mlv_dump: http://www.magiclantern.fm/forum/index.php?topic=7122.0
cr2hdr: http://www.magiclantern.fm/forum/index.php?topic=7139.0

185
convmlv.sh
View File

@ -33,6 +33,12 @@ fi
#BASIC FUNCTIONS #BASIC FUNCTIONS
#NOTE: How to use a nested array in an associative array:
# declare -a outArray=${assocArray[key]}
#NOTE: How to place nested array in an associative array:
# assocArray[key]="( \"hello\" \"world\" )". Notice the escaped internal "".
readlinkF() { #readlink -f, but works on all platforms (including mac). readlinkF() { #readlink -f, but works on all platforms (including mac).
target=$1 target=$1
@ -69,9 +75,32 @@ setPaths() { #Repends on SRC_PATH, BIN_PATH, and PYTHON. Run this function if ei
COLOR_LUTS=("${SCRIPT_LOCATION}/color-core" "${SCRIPT_LOCATION}/color-ext") #One can add more with options, but these are the defaults. COLOR_LUTS=("${SCRIPT_LOCATION}/color-core" "${SCRIPT_LOCATION}/color-ext") #One can add more with options, but these are the defaults.
DCRAW="dcraw" DCRAW="dcraw"
#new array method
#BINARY PATHS
SETTINGS[bin_mlvdump]="${SETTINGS[path_bin]}/mlv_dump" #Path to mlv_dump location.
SETTINGS[bin_rawdump]="${SETTINGS[path_bin]}/raw2dng" #Path to raw2dng location.
SETTINGS[bin_cr2hdr]="${SETTINGS[path_bin]}/cr2hdr" #Path to cr2hdr location.
SETTINGS[bin_mlv2badpixels]="${SETTINGS[path_bin]}/mlv2badpixels.sh"
SETTINGS[bin_dcraw]="dcraw"
#PYTHON PATHS
SETTINGS[bin_python_balance]="${SETTINGS[path_src]}/imgProcessing/balance.py"
SETTINGS[bin_python_sRange]="${SETTINGS[path_src]}/imgProcessing/sRange.py"
SETTINGS[bin_balance]="${SETTINGS[path_python]} ${SETTINGS[bin_python_balance]}"
SETTINGS[bin_sRange]="${SETTINGS[path_python]} ${SETTINGS[bin_python_sRange]}"
#COLOR PATHS
SETTINGS[col_lutList]="( \"${SETTINGS[path_script]}/color-core\" \"${SETTINGS[path_script]}/color-ext\" )" #One can add more with options, but these are the defaults.
} }
setDefaults() { #Set all the default global variables. Run during "parseAll". setDefaults() { #Set all the default global variables. Run during "parseAll".
#SETTINGS ARRAY GENERATION
unset SETTINGS
declare -g SETTINGS
#DEPENDENCIES #DEPENDENCIES
DEB_DEPS="imagemagick dcraw ffmpeg python3 python3-pip libimage-exiftool-perl libc6-i386" #Dependency package names (Debian). List with -K option. DEB_DEPS="imagemagick dcraw ffmpeg python3 python3-pip libimage-exiftool-perl libc6-i386" #Dependency package names (Debian). List with -K option.
UBU_DEPS="imagemagick dcraw ffmpeg python3 python3-pip libimage-exiftool-perl libc6-i386" #Dependency package names (Ubuntu). List with -K option. UBU_DEPS="imagemagick dcraw ffmpeg python3 python3-pip libimage-exiftool-perl libc6-i386" #Dependency package names (Ubuntu). List with -K option.
@ -88,6 +117,23 @@ setDefaults() { #Set all the default global variables. Run during "parseAll".
PYTHON="python" PYTHON="python"
fi fi
#new array method
SETTINGS[deps_deb]="imagemagick dcraw ffmpeg python3 python3-pip libimage-exiftool-perl libc6-i386" #Dependency package names (Debian). List with -K option.
SETTINGS[deps_ubu]="imagemagick dcraw ffmpeg python3 python3-pip libimage-exiftool-perl libc6-i386" #Dependency package names (Ubuntu). List with -K option.
SETTINGS[deps_fed]="ImageMagick dcraw ffmpeg python3 python-pip perl-Image-ExifTool glibc-devel.i686" #Dependency package names (Fedora). List with -K option.
SETTINGS[deps_brew]="imagemagick dcraw ffmpeg python3 exiftool"
SETTINGS[deps_pip]="numpy tifffile" #You dont need Pillow. Thats just to make balance.py a bit more portable.
SETTINGS[deps_man]="mlv_dump raw2dng cr2hdr mlv2badpixels.sh balance.py sRange.py color-core"
if [[ $OSTYPE == "linux-gnu" ]]; then
SETTINGS[path_python]="python3"
elif [[ $OSTYPE == "darwin11" ]]; then
SETTINGS[path_python]="python3"
else
SETTINGS[path_python]="python"
fi
#PATHS #PATHS
SCRIPT_LOCATION=$(dirname "$(readlinkF "$0")") SCRIPT_LOCATION=$(dirname "$(readlinkF "$0")")
@ -101,6 +147,18 @@ setDefaults() { #Set all the default global variables. Run during "parseAll".
OUTDIR="./raw_conv" OUTDIR="./raw_conv"
isOutGen=false isOutGen=false
#new array method
SETTINGS[path_script]=$(dirname "$(readlinkF "$0")")
SETTINGS[path_src]="$SCRIPT_LOCATION/src" #Source components will always be looked for here.
SETTINGS[path_bin]="./binaries"
SETTINGS[path_config_global]="${HOME}/convmlv.conf"
SETTINGS[path_config_local]="" #No local config by default.
setPaths #Set all the paths using the current SRC_PATH and BIN_PATH.
SETTINGS[path_output]="./raw_conv"
#OUTPUT #OUTPUT
MOVIE=false MOVIE=false
RES_IN="" RES_IN=""
@ -113,6 +171,19 @@ setDefaults() { #Set all the default global variables. Run during "parseAll".
isH264=false isH264=false
KEEP_DNGS=false KEEP_DNGS=false
#new array method
SETTINGS[develop_movie]=false
SETTINGS[develop_images]=false
RES_IN=""
SETTINGS[raw_fps]=24 #Will be read from .MLV or .RAW.
SETTINGS[img_format]="exr"
SETTINGS[img_compress]=true
SETTINGS[proxy_jpg]=false
SETTINGS[proxy_h264]=false
SETTINGS[proxy_scale]="50%"
SETTINGS[develop_keep-raw]=false
#~ COMPRESS=""
#FRAME RANGE #FRAME RANGE
FRAME_RANGE="" #UPDATED LATER WHEN FRAME # IS AVAILABLE. FRAME_RANGE="" #UPDATED LATER WHEN FRAME # IS AVAILABLE.
FRAME_START="1" FRAME_START="1"
@ -120,16 +191,47 @@ setDefaults() { #Set all the default global variables. Run during "parseAll".
RANGE_BASE="" RANGE_BASE=""
isFR=true isFR=true
#new array method
SETTINGS[frame_range]="" #UPDATED LATER WHEN FRAME # IS AVAILABLE.
SETTINGS[frame_start]="1"
SETTINGS[frame_end]=""
#~ RANGE_BASE="" #Where the raw argument frame range is spit into.
#~ isFR=true #True when frame range needn't be touched.
#RAW DEVELOPOMENT #RAW DEVELOPOMENT
HIGHLIGHT_MODE="0" HIGHLIGHT_MODE="0"
PROXY_SCALE="50%" PROXY_SCALE="50%"
DEMO_MODE="1" DEMO_MODE="1"
DEPTH="-W -6" DEPTH="-W -6"
DEPTH_OUT="-depth 16" DEPTH_OUT="-depth 16"
WAVE_NOISE="" #Used to be NOISE_REDUC. Wavelet noise reduction. WAVE_NOISE=""
FOUR_COLOR="" FOUR_COLOR=""
CHROMA_SMOOTH="--no-cs" CHROMA_SMOOTH="--no-cs"
#new array method
SETTINGS[dev_blacklevel]="2048" #MODE DETERMINE AFTER PARSE
SETTINGS[dev_satpoint]="standard" #MODE DETERMINE AFTER PARSE
#Options: standard, <any num>
SETTINGS[dcraw_satpoint]=""
SETTINGS[dcraw_high-mode]="0"
SETTINGS[dev_demosaic]="ppg" #MODE DETERMINE AFTER PARSE.
#OPTIONS: bilin, vng, ppg, ahd.
SETTINGS[dcraw_demosaic]="2"
SETTINGS[dev_csmooth]="none" #MODE DETERMINE AFTER PARSE.
#OPTIONS: none, 2x2, 3x3, 5x5
SETTINGS[dcraw_csmooth]="--no-cs"
SETTINGS[cr2hdr_csmooth]="--no-cs"
SETTINGS[mlvfs_csmooth]=""
SETTINGS[dev_depth]="16" #MODE DETERMINE AFTER PARSE.
SETTINGS[dcraw_depth]="-W -6"
SETTINGS[im_depth]="-depth 16"
SETTINGS[nr_wave]=""
SETTINGS[dcraw_four-color]=""
#COLOR MANAGEMENT #COLOR MANAGEMENT
GAMMA="1 1" #As far as dcraw is concerned, output is linear. GAMMA="1 1" #As far as dcraw is concerned, output is linear.
SPACE="5" #dcraw only outputs Linear XYZ. LUTs convert onwards. SPACE="5" #dcraw only outputs Linear XYZ. LUTs convert onwards.
@ -138,6 +240,21 @@ setDefaults() { #Set all the default global variables. Run during "parseAll".
COLOR_VF="" #Standard (~2.4) sRGB LUT by default: ${CORE_LUT}/lin_xyz--srgb_srgb.cube . This is used in VF_FILTERS COLOR_VF="" #Standard (~2.4) sRGB LUT by default: ${CORE_LUT}/lin_xyz--srgb_srgb.cube . This is used in VF_FILTERS
colorDesc="" colorDesc=""
#new array method
SETTINGS[col_gamma]="lin"
SETTINGS[dcraw_gamma]="1 1"
SETTINGS[col_gamut]="srgb"
SETTINGS[dcraw_space]="5"
#Again, the dcraw-specific stuff is assembled down the line.
#~ GAMMA="1 1" #As far as dcraw is concerned, output is linear.
#~ SPACE="5" #dcraw only outputs Linear XYZ. LUTs convert onwards.
#~ COLOR_GAMMA="lin" #STANDARD marks it such that it will correspond to the gamut
#~ COLOR_GAMUT="srgb"
#~ COLOR_VF="" #Standard (~2.4) sRGB LUT by default: ${CORE_LUT}/lin_xyz--srgb_srgb.cube . This is used in VF_FILTERS
#~ colorDesc=""
#FEATURES #FEATURES
DUAL_ISO=false DUAL_ISO=false
BADPIXELS="" BADPIXELS=""
@ -147,11 +264,28 @@ setDefaults() { #Set all the default global variables. Run during "parseAll".
useDF=false useDF=false
DARK_PROC="" DARK_PROC=""
RES_DARK="" RES_DARK=""
SETTINGS_OUTPUT=false
MK_DARK=false
DARK_OUT="" DARK_OUT=""
BLACK_LEVEL="" BLACK_LEVEL=""
#new array method
SETTINGS[dev_diso]=false
SETTINGS[dev_badpixel]=false #MODE DETERMINE AFTER PARSE
SETTINGS[path_badpixel-cust]=""
SETTINGS[dcraw_badpixel]=""
SETTINGS[cali_bias_num]=-1 #0 to index, -1 denotes not used.
SETTINGS[cali_dark_num]=-1 #0 to index, -1 denotes not used.
SETTINGS[cali_flat_num]=-1 #0 to index, -1 denotes not used.
SETTINGS[cali_bias_path]=""
SETTINGS[cali_dark_path]=""
SETTINGS[cali_flat_path]=""
SETTINGS[cali_out_path]="" #The filename (sans extension) finished calibration frames are spit out to.
#~ DARK_PROC="" #Variable where dcraw darkframe snipper can be found.
#White Balance #White Balance
WHITE="" WHITE=""
GEN_WHITE=false GEN_WHITE=false
@ -160,6 +294,12 @@ setDefaults() { #Set all the default global variables. Run during "parseAll".
isScale=false isScale=false
SATPOINT="" SATPOINT=""
WHITE=""
GEN_WHITE=false
CAMERA_WB=true
WHITE_SPD=15
isScale=false
#FFMPEG Filters #FFMPEG Filters
FFMPEG_FILTERS=false #Whether or not FFMPEG filters are going to be used. FFMPEG_FILTERS=false #Whether or not FFMPEG filters are going to be used.
V_FILTERS="" V_FILTERS=""
@ -178,6 +318,41 @@ setDefaults() { #Set all the default global variables. Run during "parseAll".
SHARP="" SHARP=""
sharpDesc="" sharpDesc=""
#new array method
SETTINGS[ffilters]=false #Whether or not FFMPEG filters are going to be used.
#~ V_FILTERS=""
#~ V_FILTERS_PROX=""
#~ FILTER_ARR=()
SETTINGS[ffilters_temp-noise]=""
#~ TEMP_NOISE="" #Temporal noise reduction.
#~ tempDesc=""
SETTINGS[ffilters_luts]="()"
#~ LUTS=() #lut3d LUT application. Supports multiple LUTs, in a chain; therefore it is an array.
#~ lutDesc=""
SETTINGS[ffilters_deshake]=""
#~ DESHAKE="" #deshake video stabilisation.
#~ deshakeDesc=""
SETTINGS[ffilters_hqnoise]=""
#~ HQ_NOISE="" #hqdn3d noise reduction.
#~ hqDesc=""
SETTINGS[ffilters_remnoise]=""
#~ REM_NOISE="" #removegrain noise reduction
#~ remDesc=""
SETTINGS[ffilters_sharp]=""
#~ SHARP=""
#~ sharpDesc=""
SETTINGS[raw_cam-name]="Unknown"
SETTINGS[raw_frames]="Unknown"
SETTINGS[raw_resolution]="Unknown"
SETTINGS[raw_iso]="Unknown"
SETTINGS[raw_aperture]="Unknown"
SETTINGS[raw_flength]="Unknown"
SETTINGS[raw_shutter]="Unknown"
SETTINGS[raw_rec-time]="Unknown"
SETTINGS[raw_rec-date]="Unknown"
SETTINGS[raw_kelvin]="Unknown"
baseSet() { #All camera attributes are reset here. baseSet() { #All camera attributes are reset here.
CAM_NAME="Unknown" CAM_NAME="Unknown"
FRAMES="Unknown" FRAMES="Unknown"
@ -214,6 +389,7 @@ source "$SRC_PATH/helpers/format.sh"
source "$SRC_PATH/programs/califrame.sh" source "$SRC_PATH/programs/califrame.sh"
source "$SRC_PATH/helpers/utility.sh" source "$SRC_PATH/helpers/utility.sh"
source "$SRC_PATH/core/parsing.sh" source "$SRC_PATH/core/parsing.sh"
source "$SRC_PATH/core/argFuncs.sh"
source "$SRC_PATH/helpers/error.sh" source "$SRC_PATH/helpers/error.sh"
source "$SRC_PATH/core/develop.sh" source "$SRC_PATH/core/develop.sh"
source "$SRC_PATH/core/proc.sh" source "$SRC_PATH/core/proc.sh"
@ -276,6 +452,9 @@ case "$PROGRAM" in
#Do the development step, using the globals that exist. #Do the development step, using the globals that exist.
develop "$ARG" develop "$ARG"
#Experimental Assoc Array
unset SETTINGS
#RESET ARGS & REPARSE OPTIONS - same as in convmlv.sh. #RESET ARGS & REPARSE OPTIONS - same as in convmlv.sh.
#Big parse/reparse, making sure global, local, command line options all override each other correctly. #Big parse/reparse, making sure global, local, command line options all override each other correctly.
set -- $INPUT_ARGS #Reset the argument input for reparsing. set -- $INPUT_ARGS #Reset the argument input for reparsing.

281
docs/MANPAGE
View File

@ -1,281 +0,0 @@
Usage:
./convmlv.sh [FLAGS] [OPTIONS] files
INFO:
A program allowing you to develop ML files into workable formats. Many useful options are exposed.
--> Defaults: Compressed 16-bit Linear EXR. 10-bit Prores4444 MOV.
--> Color Defaults: Linear (1.0) Gamma on sRGB Gamut, using Camera White Balance.
--> Acceptable Inputs: MLV, RAW (requires raw2dng), Folder containing DNGs.
--> Option Input: From command line or config file (specify with -C).
--> Forum Post: http://www.magiclantern.fm/forum/index.php?topic=16799.
--> A note: BE CAREFUL WITH OPTIONS. Wrong values will give very strange errors. Read this page well!!
It's as simple or complex as you need it to be: 'convmlv -m <mlvfile>.mlv' is enough for good-looking output!
VERSION: 2.0.1
MANUAL DEPENDENCIES:
Place these in RES_PATH (see OPTIONS, BASIC). Keep in mind you also need dist. and pip packages.
--> See 'Dist Deps' and 'Python Deps'
-- mlv_dump: Required binary. http://www.magiclantern.fm/forum/index.php?topic=7122.0
-- color-core: Required folder of LUTs. See convmlv repository.
-- sRange.py: Required script. See convmlv repository.
-- raw2dng: For DNG extraction from RAW. http://www.magiclantern.fm/forum/index.php?topic=5404.0
-- mlv2badpixels.sh: For bad pixel removal. https://bitbucket.org/daniel_fort/ml-focus-pixels/src
-- cr2hdr: For Dual ISO Development. Two links: http://www.magiclantern.fm/forum/index.php?topic=16799.0
-- balance.py: For Auto White Balance. See convmlv repository.
-- color-ext: Extra LUTs, providing more color resources. See convmlv repository.
OPTIONS, BASIC:
-v, --version version - Print out version string.
-h, --help help - Print out this help page.
-C, --config config - Designates config file to use.
-o, --outdir <path> OUTDIR - The path in which files will be placed.
-P, --res-path <path> RES_PATH - The path in which all manual dependencies are looked for.
--> Default: Current Directory.
--dcraw <path> DCRAW - The path to dcraw.
--mlv-dump <path> MLV_DUMP - The path to mlv_dump.
--raw-dump <path> RAW_DUMP - The path to raw2dng.
--badpixels <path> MLV_BP - The path to mlv2badpixels.sh (by dfort).
--cr-hdr <path> CR_HDR - The path to cr2hdr.
--srange <path> SRANGE - The path to sRange.py.
--balance <path> BAL - The path to balance.py.
--python <path> PYTHON - The path or command used to invoke Python 3. Default is python3 on Linux, python on Mac.
-T, --threads [int] THREADS - Override amount of utilized process threads. Default is MAX - 1.
OPTIONS, OUTPUT:
-i IMAGE - Will output image sequence.
-t [0:3] IMG_FMT - Image output format.
--> 0: EXR (default), 1: TIFF, 2: PNG, 3: Cineon (DPX)."
-m MOVIE - Will output a Prores4444 file.
-p [0:3] PROXY - Create proxies alongside main output.
--> 0: No proxies (Default). 1: H.264 proxy. 2: JPG proxy sequence. 3: Both.
--> JPG proxy will always be in sRGB Gamma/sRGB Gamut. H.264 proxy is color managed.
--> JPG proxy *won't* be developed w/o IMAGE. H.264 proxy *will* be developed no matter what, if specified here.
--> Why? JPG is for potential use in editing. H.264 is for a quick visual preview of convmlv's output.
-s [0%:100%] PROXY_SCALE - the size, in %, of the proxy output.
--> Default: 50%.
-k KEEP_DNGS - Specify if you want to keep the DNG files.
--> Run convmlv on the top level folder of former output to reuse saved DNGs from that run!
-r <start>-<end> FRAME_RANGE - Specify to process an integer frame range.
--> You may use the characters 's' and 'e', such that s = start frame, e = end frame.
--> Indexed from 0 to (# of frames - 1). Develops from 1 to ($ of frames)
--> A single number may be writted to develop that single frame.
--> DO NOT try to reuse DNGs while developing a larger frame range.
--uncompress UNCOMP - Turns off lossless image compression. Otherwise:
--> TIFF: ZIP, EXR: PIZ, PNG: lvl 0, DPX: RLE.
OPTIONS, RAW DEVELOPMENT:
-d [0:3] DEMO_MODE - Demosaicing algorithm. Higher modes are slower + better.
--> 0: Bilinear. 1: VNG (default). 2: PPG. 3: AHD.
-f FOUR_COLOR - Interpolate as RGBG. Fixes weirdness with VNG/AHD, at the cost of sharpness.
-H [0:9] HIGHLIGHT_MODE - Highlight management options.
--> 0: White, clipped highlights. 1: Unclipped but colored highlights. 2: The defail of 1, but adjusted to grey.
--> 3-9: Highlight reconstruction. Can cause flickering. Start at 5, then adjust to color (down) or to white (up).
-c [0:3] CHROMA_SMOOTH - Apply shadow/highlight chroma smoothing to the footage.
--> 0: None (default). 1: 2x2. 2: 3x3. 3: 5x5.
--> MLV input Only.
-n [int] WAVE_NOISE - Apply wavelet denoising.
--> Default: None. Subtle: 25. Medium: 50. Strong: 125.
-N <A>-<B> TEMP_NOISE - Apply temporal denoising.
--> A: 0 to 0.3. B: 0 to 5. A reacts to abrupt noise (splotches), B reacts to noise over time (fast motion causes artifacts).
--> Subtle: 0.03-0.04. High: 0.15-0.04. High, Predictable Motion: 0.15-0.07
-Q [i-i:i-i] HQ_NOISE - Apply 3D denoising filter.
--> In depth explanation: https://mattgadient.com/2013/06/29/in-depth-look-at-de-noising-in-handbrake-with-imagevideo-examples/ .
--> Spacial/Temporal (S/T). S will soften/blur/smooth, T will remove noise without doing that but may create artifacts.
--> Luma/Chroma (L/C). L is the detail, C is the color. Each one's denoising may be manipulated Spacially or Temporally.
--> Option Value: <LS>-<CS>:<LT>-<CT>
--> Weak: 2-1:2-3. Medium: 3-2:2-3. Strong: 7-7:5-5
--> DONT combine with TEMP_NOISE.
-O [i-i-i-i] REM_NOISE - Yet another spatial denoiser, with 4 choices of 24 modes.
--> See https://ffmpeg.org/ffmpeg-filters.html#removegrain for list of modes.
--> Option Value: <mode1>-<mode2>-<mode3>-<mode4>
--> I truly cannot tell you what values will be helpful to you; there are too many... Look at the link!
--shallow SHALLOW - Output smaller, 8-bit files.
--> Read why this is a bad idea: http://www.cambridgeincolour.com/tutorials/bit-depth.htm
OPTIONS, COLOR:
-g [0:4] GAMMA - Output gamma. A curve applied to the output, for easier viewing/grading.
--> 0: Standard (Around 2.2). 1: Linear (Default).
--> Requires color-ext: 2: Cineon. 3: C-Log2 4: S-Log3
--> "Standard" grades to the gamut specification, and to 2.2 if that's not given.
-G [0:6] GAMUT - Output gamut. The range of colors that can exist in the output.
--> 0: sRGB (Default). 1: Adobe RGB. 2: Rec.709. 3: XYZ (Always Linear Gamma).
--> Requires color-ext: 4: Rec2020 5: DCI-P3 6: Sony S-Gamut3.cine
-w [0:2] WHITE - This is a modal white balance setting.
--> 0: Auto WB (requires balance.py). 1: Camera WB (default). 2: No Change.
--> AWB uses the Grey's World algorithm.
-A [i:i:i:i] SHARP - Lets you sharpen, or blur, your footage.
--> BE CAREFUL. Wrong values will give you strange errors.
--> Size/Strength (S/T). S is the size of the sharpen/blur effect, T is the strength of the sharpen/blur effect.
--> Luma/Chroma (L/C). L is the detail, C is the color. Luma sharpening more effective.
--> Option Value: <LS>:<LT>:<CS>:<CT>
--> LS and CS must be ODD, between 3 and 63. Negative LT/CT values blur, while positive ones sharpen.
--> Strong Sharp: 7:3:7:3 Strong Blur: 7,-3:7,-3. Medium Sharp: 5:1:3:0
-l <path> LUT - Specify a LUT to apply after Color Management.
--> Supports cube, 3dl, dat, m3d.
--> Specify -l multiple times, to apply multiple LUTs in sequence.
-S [int] SATPOINT - Specify the 14-bit uint saturation point of your camera. You don't usually need to.
--> Worth setting globally, as it's a per-camera setting. Must be correct for highlight reconstruction/unclipped highlights.
--> Lower from 15000 if -H1 yields purple highlights, until they turn white.
--> You can determine the optimal value using the max pixel value of 'dcraw -D -j -4 -T'.
--white-speed [int] WHITE_SPD - Manually specify samples used to calculate AWB.
--allow-white-clip WHITE_CLIP - Let the White Balance multipliers clip.
OPTIONS, FEATURES:
-D DESHAKE - Auto-stabilize the video using ffmpeg's "deshake" module.
--> You may wish to crop/scale the output later, to avoid edge artifacts.
-u DUAL_ISO - Process as dual ISO.
--> Requires cr2hdr.
-b BADPIXELS - Fix focus pixels issue using dfort's script.
--> Requires mlv2badpixels.sh.
-a <path> BADPIXEL_PATH - Use your own .badpixels file. Does NOT require mlv2badpixels.sh
--> How to: http://www.dl-c.com/board/viewtopic.php?f=4&t=686
-F <path> DARKFRAME - This is the path to a "dark frame MLV"; effective for noise reduction.
--> How to: Record 5 sec w/lens cap on & same settings as footage. Pass MLV in here.
--> If the file extension is '.darkframe', the file will be used as a preaveraged dark frame.
-R <path> darkframe_output - Specify to create a .darkframe file from passed in MLV.
--> Usage: 'convmlv -R <path> <input>.MLV'
--> Averages <input>.MLV to create <path>.darkframe.
--> THE .darkframe EXTENSION IS ADDED FOR YOU.
OPTIONS, INFO:
-q settings - Output MLV settings.
-K [0:3] Dist Deps - Output package dependecies, for use with common package managers.
--> 0: Debian, 1: Ubuntu, 2: Fedora, 3: Homebrew (Mac)
--> Deps Install (Debian): sudo apt-get install $(./convmlv.sh -K 0)
--> Deps Install (Ubuntu): sudo apt-get install $(./convmlv.sh -K 1)
--> Deps Install (Fedora): sudo yum install $(./convmlv.sh -K 2)
--> Deps Install (Homebrew Mac): brew install $(./convmlv.sh -K 3)
-Y Python Deps - Lists Python dependencies. Works directly with pip.
-->Install (Cross-Platform): sudo python3 -m pip install $ (./convmlv -Y)
-M Manual Deps - Lists manual dependencies, which must be downloaded by hand.
--> Manually place all in RES_PATH. See http://www.magiclantern.fm/forum/index.php?topic=16799.0 .
COLOR MANAGEMENT:
INTRO Images aren't simple. They are often stored, processed, and viewed as a result of complex transformations usually called,
in applications, color management. Understanding it is required as a colourist, and encouraged as a DPs and Cinematographers.
--> Intro: http://www.cambridgeincolour.com/tutorials/color-management1.htm
--> Understanding Gamma: http://www.cambridgeincolour.com/tutorials/gamma-correction.htm
--> Color Spaces: http://www.cambridgeincolour.com/tutorials/color-spaces.htm
--> Conversions: http://www.cambridgeincolour.com/tutorials/color-space-conversion.htm
--> Monitor Calibration: http://www.cambridgeincolour.com/tutorials/monitor-calibration.htm
PIPELINE convmlv is a color managed application, designed to retain quality from RAW footage:
-- mlv_dump writes camera-specific color transformation matrices as metadata in developed DNGs.
--> This defines the camera's gamut.
-- dcraw applies these matrices, then transforms the newly developed image to the super-wide XYZ colorspace.
--> All color detail is preserved, and if now in a well-defined colorspace.
--> No gamma has been applied - the image is now Linear XYZ.
-- ffmpeg applies the specified (up to) x64 resolution 3D LUTs, in .cube format.
--> Use -g and -G to specify Gamma/Gamut combinations.
--> The output employs DATA, not LEGAL, values. The resulting image is flatter, but retains all shadow/highlight detail.
--> -l specified LUTs are applied afterwards.
3D LUTS The included LUTs, found in the convmlv repository, are key to convmlv's color management solution:
-- color-core: The required LUTs, including sRGB (default), Adobe RGB, and Rec709 in Standard and Linear gamma.
-- color-ext: Optional LUTs, including Rec2020, DCI-P3, etc. in Standard/Linear gammas, but also in Log formats.
Create Your Own LUTs using LUTCalc, for any grading format output: https://cameramanben.github.io/LUTCalc/ (watch his tutorials).
--> Note that convmlv only accepts up to 64x64x64 LUTs. You can resize LUTs using pylut (https://pypi.python.org/pypi/pylut).
--> The pylut command to resize is 'pylut <yourx65lut>.cube --resize 64'. Alternatively, you can use pylut from Python (2X only).
--> I reccommend Legal --> Data LUTs, as this conserves shadow/highlight detail for grading. Legal --> Legal looks better, but with detail loss.
CONFIG FILE:
Config files, another way to specify options, can save you time & lend you convenience in production situations.
GLOBAL: /home/sofus/convmlv.conf
LOCAL: Specify -C/--config.
SYNTAX:
Most options listed above have an uppercased VARNAME, ex. OUTDIR. You can specify such options in config files, as such:
<VARNAME> <VALUE>
One option per line only. Indentation by tabs or spaces is allowed, but not enforced.
Comments Lines starting with # are comments.
You may name a config using:
CONFIG_NAME <name>
Flags If the value is a true/false flag (ex. IMAGE), simply specifying VARNAME is enough. There is no VALUE.
OPTION ORDER OF PRECEDENCE Options override each other as such:
-LOCAL options overwrite GLOBAL options.
-COMMAND LINE options overwrite LOCAL & GLOBAL options.
-FILE SPECIFIC options overwrite ALL ABOVE options.
File-Specific Block: A LOCAL config file lets you specify options for specific input names:
/ <TRUNCATED INPUTNAME>
...options here will only be
*
You must use the truncated (no .mlv or .raw) input name after the /. Nested blocks will fail.
With a single config file, you can control the development options of multiple inputs as specifically and/or generically
as you want. Batch developing everything can then be done with a single, powerful commmand.
Contact me with any feedback or questions at convmlv@sofusrose.com, PM me (so-rose) on the ML forums, or post on the thread!

4
docs/buildDocs.sh
View File

@ -7,4 +7,6 @@ DDIR="$ROOT_DIR/docs"
$ROOT_DIR/convmlv.sh -h | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" > $DDIR/MANPAGE $ROOT_DIR/convmlv.sh -h | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" > $DDIR/MANPAGE
pdflatex $DDIR/docs.tex #> /dev/null pdflatex $DDIR/docs.tex #> /dev/null
pdflatex $DDIR/docs.tex #> /dev/null #Compile twice to get TOC to show. pdflatex $DDIR/docs.tex #> /dev/null #Compile twice to get TOC to show.
rm -f $DDIR/docs.aux $DDIR/docs.log $DDIR/docs.out $DDIR/docs.toc $DDIR/texput.log > /dev/null mv ./docs.pdf $DDIR
rm -f ./docs.aux ./docs.log ./docs.out ./docs.toc ./texput.log > /dev/null

BIN
docs/docs.pdf
View File

Binary file not shown.

54
geany-project 100644
View File

@ -0,0 +1,54 @@
[editor]
line_wrapping=false
line_break_column=72
auto_continue_multiline=true
[file_prefs]
final_new_line=true
ensure_convert_new_lines=false
strip_trailing_spaces=false
replace_tabs=false
[indentation]
indent_width=4
indent_type=1
indent_hard_tab_width=8
detect_indent=false
detect_indent_width=false
indent_mode=2
[project]
name=convmlv
base_path=/home/sofus/subhome/src/convmlv
[long line marker]
long_line_behaviour=1
long_line_column=72
[files]
current_page=4
FILE_NAME_0=14583;Sh;0;EUTF-8;1;1;0;%2Fhome%2Fsofus%2Fsubhome%2Fsrc%2Fconvmlv%2Fsrc%2Fcore%2Fdevelop.sh;0;4
FILE_NAME_1=3473;Sh;0;EUTF-8;1;1;0;%2Fhome%2Fsofus%2Fsubhome%2Fsrc%2Fconvmlv%2Fsrc%2Fcore%2Fparsing.sh;0;4
FILE_NAME_2=1070;Sh;0;EUTF-8;1;1;0;%2Fhome%2Fsofus%2Fsubhome%2Fsrc%2Fconvmlv%2Fsrc%2Fcore%2FargFuncs.sh;0;4
FILE_NAME_3=3159;Sh;0;EUTF-8;1;1;0;%2Fhome%2Fsofus%2Fsubhome%2Fsrc%2Fconvmlv%2Fconvmlv.sh;0;4
FILE_NAME_4=8629;Sh;0;EUTF-8;1;1;0;%2Fhome%2Fsofus%2Fsubhome%2Fsrc%2Fconvmlv%2Fsrc%2Fcore%2Fproc.sh;0;4
FILE_NAME_5=2653;Sh;0;EUTF-8;1;1;0;%2Fhome%2Fsofus%2Fsubhome%2Fsrc%2Fconvmlv%2Fsrc%2Fdocumentation%2Fhelp.sh;0;4
FILE_NAME_6=0;Sh;0;EUTF-8;1;1;0;%2Fhome%2Fsofus%2Fsubhome%2Fsrc%2Fconvmlv%2Fsrc%2Fdocumentation%2FTODO.sh;0;4
FILE_NAME_7=0;Sh;0;EUTF-8;1;1;0;%2Fhome%2Fsofus%2Fsubhome%2Fsrc%2Fconvmlv%2Fsrc%2Fhelpers%2Ferror.sh;0;4
FILE_NAME_8=0;Sh;0;EUTF-8;1;1;0;%2Fhome%2Fsofus%2Fsubhome%2Fsrc%2Fconvmlv%2Fsrc%2Fhelpers%2Fformat.sh;0;4
FILE_NAME_9=0;Sh;0;EUTF-8;1;1;0;%2Fhome%2Fsofus%2Fsubhome%2Fsrc%2Fconvmlv%2Fsrc%2Fhelpers%2Fplatform.sh;0;4
FILE_NAME_10=2653;Sh;0;EUTF-8;1;1;0;%2Fhome%2Fsofus%2Fsubhome%2Fsrc%2Fconvmlv%2Fsrc%2Fhelpers%2Futility.sh;0;4
FILE_NAME_11=0;Sh;0;EUTF-8;1;1;0;%2Fhome%2Fsofus%2Fsubhome%2Fsrc%2Fconvmlv%2Fsrc%2FimgProcessing%2FimgMath.sh;0;4
FILE_NAME_12=0;Sh;0;EUTF-8;1;1;0;%2Fhome%2Fsofus%2Fsubhome%2Fsrc%2Fconvmlv%2Fsrc%2Fprograms%2Fcaliframe.sh;0;4
FILE_NAME_13=1;Sh;0;EUTF-8;1;1;0;%2Fhome%2Fsofus%2Fsubhome%2Fsrc%2Fconvmlv%2Fsrc%2Fprograms%2FshotSettings.sh;0;4
[VTE]
last_dir=/home/sofus
[prjorg]
source_patterns=*.c;*.C;*.cpp;*.cxx;*.c++;*.cc;*.m;
header_patterns=*.h;*.H;*.hpp;*.hxx;*.h++;*.hh;
ignored_dirs_patterns=.*;CVS;
ignored_file_patterns=*.o;*.obj;*.a;*.lib;*.so;*.dll;*.lo;*.la;*.class;*.jar;*.pyc;*.mo;*.gmo;
generate_tag_prefs=0
external_dirs=

13
mkrelease.sh
View File

@ -3,16 +3,16 @@
VERSION=$(echo "$(./convmlv.sh -v)" | sed -e 's/\./\_/g') VERSION=$(echo "$(./convmlv.sh -v)" | sed -e 's/\./\_/g')
#HOW TO USE (Linux and Mac): #HOW TO USE (Linux and Mac):
# 1. Update version above. # 1. Make sure everything is up to date.
# 2. Put mlv2badpixels.sh, mlv_dump, raw2dng, and cr2hdr into a "binaries" folder in the repository. # 2. Put mlv2badpixels.sh, mlv_dump, raw2dng, and cr2hdr into the "binaries" folder in the repository.
# 3. Run this script, with one argument representing the path to the binaries. # 3. A release tarball will automatically be created in "release" in the repository.
# 4. A release tarball will automatically be created in "release" in the repository.
## It's reccommended that BINPATH is a folder in REP_PATH. ## It's reccommended that BINPATH is a folder in REP_PATH.
REP_PATH="$(pwd)" REP_PATH="$(pwd)"
BINPATH="${REP_PATH}/binaries" BINPATH="${REP_PATH}/binaries"
SRCPATH="${REP_PATH}/src" SRCPATH="${REP_PATH}/src"
DOCPATH="${REP_PATH}/docs"
RELEASE="${REP_PATH}/release" RELEASE="${REP_PATH}/release"
mkdir -p "$RELEASE" mkdir -p "$RELEASE"
@ -25,5 +25,10 @@ else
echo "Platform not yet supported! Contact me at contact@sofusrose.com." echo "Platform not yet supported! Contact me at contact@sofusrose.com."
fi fi
$DOCPATH/cleanDocs.sh > /dev/null
$DOCPATH/buildDocs.sh > /dev/null
cd $REP_PATH cd $REP_PATH
tar -czvf $RELEASE/convmlv-${VERSION}-${PLATFORM}.tar.gz binaries/ src/ CHANGELOG licence convmlv.sh color-core/ color-ext DEPENDENCIES docs/MANPAGE docs/docs.pdf docs/workflow.txt configs/* tar -czvf $RELEASE/convmlv-${VERSION}-${PLATFORM}.tar.gz binaries/ src/ CHANGELOG licence convmlv.sh color-core/ color-ext DEPENDENCIES docs/MANPAGE docs/docs.pdf docs/workflow.txt configs/*
$DOCPATH/cleanDocs.sh > /dev/null

365
src/core/argFuncs.sh 100644
View File

@ -0,0 +1,365 @@
argOUTDIR() {
val="$1"
OUTDIR="$val"
SETTINGS[path_output]="$val"
}
argBIN_PATH() {
val="$1"
BIN_PATH=="$val"
SETTINGS[path_bin]=="$val"
}
argDCRAW() {
val="$1"
DCRAW="$val"
SETTINGS[bin_dcraw]="$val"
}
argMLV_DUMP() {
val="$1"
MLV_DUMP="$val"
SETTINGS[bin_mlvdump]="$val"
}
argRAW_DUMP() {
val="$1"
RAW_DUMP="$val"
SETTINGS[bin_rawdump]="$val"
}
argMLV_BP() {
val="$1"
MLV_BP="$val"
}
argCR_HDR() {
val="$1"
CR_HDR="$val"
}
argPYTHON() {
val="$1"
PYTHON="$val"
}
argTHREADS() {
val="$1"
THREADS="$val"
}
argIMAGE() {
IMAGES=true
}
argIMG_FMT() {
val="$1"
case ${val} in
"0") IMG_FMT="exr"
;;
"1") IMG_FMT="tiff"
;;
"2") IMG_FMT="png"
;;
"3") IMG_FMT="dpx"
;;
*) invOption "Invalid Image Format Choice: ${mode}"
;;
esac
}
argMOVIE() {
MOVIE=true
}
argPROXY() {
val="$1"
case ${val} in
"0") isJPG=false; isH264=false
;;
"1") isJPG=false; isH264=true
;;
"2") isJPG=true; isH264=false
;;
"3") isJPG=true; isH264=true
;;
*) invOption "Invalid Proxy Choice: ${PROXY}"
;;
esac
}
argPROXY_SCALE() {
val="$1"
PROXY_SCALE=val
proxy_num=`echo "$PROXY_SCALE" | cut -d'%' -f 1`
if [[ ! ( ($proxy_num -le 100 && $proxy_num -ge 5) && $proxy_num =~ ^-?[0-9]+$ ) ]];then
invOption "Invalid Proxy Scale: ${PROXY_SCALE}"
fi
}
argKEEP_RAWS() {
val="$1"
KEEP_DNGS=true
}
argFRAME_RANGE() {
val="$1"
RANGE_BASE="$val"
isFR=false
}
argUNCOMP() {
isCOMPRESS=false
}
argDEMO_MODE() {
val="$1"
DEMO_MODE="$val"
}
argHIGHLIGHT_MODE() {
val="$1"
HIGHLIGHT_MODE="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}
argOUTDIR() {
val="$1"
OUTDIR="$val"
}

6
src/core/develop.sh
View File

@ -123,9 +123,9 @@ develop() {
#Establish Basic Directory Structure. #Establish Basic Directory Structure.
OUTDIR=$(readlinkF "$OUTDIR") OUTDIR=$(readlinkF "$OUTDIR")
if [ $OUTDIR != $PWD ] && [ $isOutGen == false ]; then if [ $OUTDIR != $PWD ]; then
mkdir -p $OUTDIR #NO RISKS. WE REMEMBER THE LUT.py. RIP ad-hoc HALD LUT implementation :'( . mkdir -p $OUTDIR
isOutGen=true #NO RISKS. WE REMEMBER THE LUT.py. RIP ad-hoc HALD LUT implementation :'( .
fi fi
local FILE="${OUTDIR}/${TRUNC_ARG}" local FILE="${OUTDIR}/${TRUNC_ARG}"

14
src/core/parsing.sh
View File

@ -2,10 +2,12 @@
#desc: All config and command line parsing happens here. #desc: All config and command line parsing happens here.
#MAIN FUNCTIONS
parseConf() { parseConf() {
file=$1 #The File to Parse local file=$1 #The File to Parse
argOnly=$2 #If true, will only use file-specific blocks. If false, will ignore file-specific blocks. local argOnly=$2 #If true, will only use file-specific blocks. If false, will ignore file-specific blocks.
CONFIG_NAME="None" local CONFIG_NAME="None"
if [[ -z $file ]]; then return; fi if [[ -z $file ]]; then return; fi
if [[ ! -f $file ]]; then return; fi if [[ ! -f $file ]]; then return; fi
@ -46,9 +48,7 @@ parseConf() {
;; ;;
"MLV_BP") MLV_BP=`echo "${line}" | cut -d$' ' -f2` "MLV_BP") MLV_BP=`echo "${line}" | cut -d$' ' -f2`
;; ;;
"SRANGE") CR_HDR=`echo "${line}" | cut -d$' ' -f2` "CR_HDR") CR_HDR=`echo "${line}" | cut -d$' ' -f2`
;;
"BAL") PYTHON_SRANGE=`echo "${line}" | cut -d$' ' -f2`; setPaths
;; ;;
"PYTHON") PYTHON=`echo "${line}" | cut -d$' ' -f2`; setPaths "PYTHON") PYTHON=`echo "${line}" | cut -d$' ' -f2`; setPaths
;; ;;
@ -96,7 +96,7 @@ parseConf() {
proxy_num=`echo "$PROXY_SCALE" | cut -d'%' -f 1` proxy_num=`echo "$PROXY_SCALE" | cut -d'%' -f 1`
if [[ ! ( ($proxy_num -le 100 && $proxy_num -ge 5) && $proxy_num =~ ^-?[0-9]+$ ) ]]; then invOption "Invalid Proxy Scale: ${PROXY_SCALE}"; fi if [[ ! ( ($proxy_num -le 100 && $proxy_num -ge 5) && $proxy_num =~ ^-?[0-9]+$ ) ]]; then invOption "Invalid Proxy Scale: ${PROXY_SCALE}"; fi
;; ;;
"KEEP_DNGS") KEEP_DNGS=true "KEEP_RAWS") KEEP_DNGS=true
;; ;;
"FRAME_RANGE") RANGE_BASE=`echo "${line}" | cut -d$' ' -f2`; isFR=false "FRAME_RANGE") RANGE_BASE=`echo "${line}" | cut -d$' ' -f2`; isFR=false
;; ;;

2
src/documentation/help.sh
View File

@ -52,8 +52,6 @@ $(head "OPTIONS, BASIC:")
--raw-dump <path> $(cVal RAW_DUMP) - The path to raw2dng. --raw-dump <path> $(cVal RAW_DUMP) - The path to raw2dng.
--badpixels <path> $(cVal MLV_BP) - The path to mlv2badpixels.sh (by dfort). --badpixels <path> $(cVal MLV_BP) - The path to mlv2badpixels.sh (by dfort).
--cr-hdr <path> $(cVal CR_HDR) - The path to cr2hdr. --cr-hdr <path> $(cVal CR_HDR) - The path to cr2hdr.
--srange <path> $(cVal SRANGE) - The path to sRange.py.
--balance <path> $(cVal BAL) - The path to balance.py.
--python <path> $(cVal PYTHON) - The path or command used to invoke Python 3. Default is python3 on Linux, python on Mac. --python <path> $(cVal PYTHON) - The path or command used to invoke Python 3. Default is python3 on Linux, python on Mac.
-T, --threads [int] $(cVal THREADS) - Override amount of utilized process threads. Default is MAX - 1. -T, --threads [int] $(cVal THREADS) - Override amount of utilized process threads. Default is MAX - 1.

8
src/helpers/utility.sh
View File

@ -81,3 +81,11 @@ runSim() {
cat $PIPE | $cmd1 & $cmdOrig | tee $PIPE | $cmd2 #The magic of simultaneous execution ^_^ cat $PIPE | $cmd1 & $cmdOrig | tee $PIPE | $cmd2 #The magic of simultaneous execution ^_^
#~ cat $PIPE | tr 'e' 'a' & echo 'hello' | tee $PIPE | tr 'e' 'o' #The magic of simultaneous execution ^_^ #~ cat $PIPE | tr 'e' 'a' & echo 'hello' | tee $PIPE | tr 'e' 'o' #The magic of simultaneous execution ^_^
} }
strArr() {
#usage: strArr arg1 arg2 ...
#desc: Formats the args into a string ready to insert into an associative array.
#return: A string ready to insert into an associative array.
echo "TO BE WRITTEN"
}