Treezer Goode

The Curious Tale of a Wood that Needed the Trees

Thomas P. Harte

Saturday, 2022-06-04

\begin{equation*} A = \min\left(1, {\tilde\pi(\theta')\,q(\theta^{(t)}\mid\theta') \over \tilde\pi(\theta^{(t)})\,q(\theta'\mid\theta^{(t)})} \right) \end{equation*}

cmd spark start cluster --instance=instance-1 --quietly

/ms/dist/python/PROJ/bin/python

/ms/dist/R/PROJ/bin/R

cmd spark start cluster --instance=instance-1 --quietly

Disclaimer

Thomas P. Harte ("the Author") is providing this presentation and its contents ("the Content") for educational purposes only at the R in Finance Conference, Chicago, IL (2022-06-04). The Author is not a registered investment advisor and the Author does not purport to offer investment advice nor business advice. The opinions expressed in the Content are solely those of the Author, and do not necessarily represent the opinions of the Author's employer, nor any organization, committee or other group with which the Author is affiliated.

THE AUTHOR SPECIFICALLY DISCLAIMS ANY PERSONAL LIABILITY, LOSS OR RISK INCURRED AS A CONSEQUENCE OF THE USE AND APPLICATION, EITHER DIRECTLY OR INDIRECTLY, OF THE CONTENT. THE AUTHOR SPECIFICALLY DISCLAIMS ANY REPRESENTATION, WHETHER EXPLICIT OR IMPLIED, THAT APPLYING THE CONTENT WILL LEAD TO SIMILAR RESULTS IN A BUSINESS SETTING. THE RESULTS PRESENTED IN THE CONTENT ARE NOT NECESSARILY TYPICAL AND SHOULD NOT DETERMINE EXPECTATIONS OF FINANCIAL OR BUSINESS RESULTS.

The Noble

Command Line Interface

Address to the Nation

on

The State of the CLI

Depends on the OS/language

But…

A bit of a mess

Jackson Pollock Number 18 (1950)—oil and enamel on masonite
Guggenheim Museum, New York

Depends on the OS/language

GNU
Python
Java
Scala
C++
R?

GNU

# a pseudo-standard
cmd --help
cmd --version

GNU: Git

git --help | head -5


usage: git [--version] [--help] [-C <path>] [-c <name>=<value>]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p | --paginate | -P | --no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]

Observation: DSL

Domain-specific language ("DSL"): The language in which the DSL is written or presented
Host language: The language in which the DSL is executed or processed

GNU

# a pseudo-standard
cmd --help
cmd --version

# subcommands
git init
git add
git commit -am
git log

GNU

git --help


usage: git [--version] [--help] [-C ] [-c =]
           [--exec-path[=]] [--html-path] [--man-path] [--info-path]
           [-p | --paginate | -P | --no-pager] [--no-replace-objects] [--bare]
           [--git-dir=] [--work-tree=] [--namespace=]
            []

These are common Git commands used in various situations:

start a working area (see also: git help tutorial)
   clone             Clone a repository into a new directory
   init              Create an empty Git repository or reinitialize an existing one

work on the current change (see also: git help everyday)
   add               Add file contents to the index
   mv                Move or rename a file, a directory, or a symlink
   restore           Restore working tree files
   rm                Remove files from the working tree and from the index
   sparse-checkout   Initialize and modify the sparse-checkout

examine the history and state (see also: git help revisions)
   bisect            Use binary search to find the commit that introduced a bug
   diff              Show changes between commits, commit and working tree, etc
   grep              Print lines matching a pattern
   log               Show commit logs
   show              Show various types of objects
   status            Show the working tree status

grow, mark and tweak your common history
   branch            List, create, or delete branches
   commit            Record changes to the repository
   merge             Join two or more development histories together
   rebase            Reapply commits on top of another base tip
   reset             Reset current HEAD to the specified state
   switch            Switch branches
   tag               Create, list, delete or verify a tag object signed with GPG

collaborate (see also: git help workflows)
   fetch             Download objects and refs from another repository
   pull              Fetch from and integrate with another repository or a local branch
   push              Update remote refs along with associated objects

'git help -a' and 'git help -g' list available subcommands and some
concept guides. See 'git help ' or 'git help '
to read about a specific subcommand or concept.
See 'git help git' for an overview of the system.

GNU: Git subcommands

git commit --help | head -13


GIT-COMMIT(1)                                                   Git Manual

NAME
       git-commit - Record changes to the repository

SYNOPSIS
       git commit [-a | --interactive | --patch] [-s] [-v] [-u<mode>] [--amend]
                  [--dry-run] [(-c | -C | --fixup | --squash) <commit>]
                  [-F <file> | -m <msg>] [--reset-author] [--allow-empty]
                  [--allow-empty-message] [--no-verify] [-e] [--author=<author>]
                  [--date=<date>] [--cleanup=<mode>] [--[no-]status]
                  [-i | -o] [--pathspec-from-file=<file> [--pathspec-file-nul]]
                  [-S[<keyid>]] [--] [<pathspec>...]

Observation: Recursion

Web-page recursion

GNU: Git subcommand

git commit --help | head -13


GIT-COMMIT(1)                                                   Git Manual

NAME
       git-commit - Record changes to the repository

SYNOPSIS
       git commit [-a | --interactive | --patch] [-s] [-v] [-u<mode>] [--amend]
                  [--dry-run] [(-c | -C | --fixup | --squash) <commit>]
                  [-F <file> | -m <msg>] [--reset-author] [--allow-empty]
                  [--allow-empty-message] [--no-verify] [-e] [--author=<author>]
                  [--date=<date>] [--cleanup=<mode>] [--[no-]status]
                  [-i | -o] [--pathspec-from-file=<file> [--pathspec-file-nul]]
                  [-S[<keyid>]] [--] [<pathspec>...]

Observation: Treezer Goode

Sarah Zielenski ScienceNews for Students (2020-04-22)

Þ Auld PyÞone

Python…

The way it used to be

Python: `argparse`

"Batteries included"
DSL?

Python: subcommands with `argparse`

cmd query host host_name
cmd query profile profile_name
cmd query environment environment_name
cmd add host host_name
cmd add profile profile_name
cmd add environment environment_name
cmd update host host_name

Python: subcommands with `argparse` (muchos fun!)

import argparse
from mock import Mock

parser           = argparse.ArgumentParser()
subparsers       = parser.add_subparsers()

query_group      = subparsers.add_parser('query')
add_group        = subparsers.add_parser('add')
update_group     = subparsers.add_parser('update')

### query
subparsers       = query_group.add_subparsers()
#   query host
host_query       = subparsers.add_parser('host')
#   query host host_name
host_query.add_argument('host_name')
host_query.set_defaults(func=m.query_host)

#   query profile
profile_query     = subparsers.add_parser('profile')
#   query profile profile_name
profile_query.add_argument('profile_name')
profile_query.set_defaults(func=m.query_profile)

#   query environment
environment_query = subparsers.add_parser('environment')
#   query environment environment_name
environment_query.add_argument('environment_name')
environment_query.set_defaults(func=m.query_environment)

### add
subparsers        = add_group.add_subparsers()
#   add host
host_add          = subparsers.add_parser('host')
#   add host host_name
host_add.add_argument('host_name')
host_add.set_defaults(func=m.add_host)

#   add profile
profile_add       = subparsers.add_parser('profile')
#   add profile profile_name
profile_add.add_argument('profile_name')
profile_add.set_defaults(func=m.add_profile)

#   add environment
environment_add   = subparsers.add_parser('environment')
#   add environment environment_name
environment_add.add_argument('environment_name')
environment_add.set_defaults(func=m.add_environment)

### update
subparsers        = update_group.add_subparsers()
#   update host
host_update       = subparsers.add_parser('host')
#   update host host_name
host_update.add_argument('host_name')
host_update.set_defaults(func=m.update_host)

profile_update    = subparsers.add_parser('profile')
profile_update.add_argument('profile_name')
profile_update.set_defaults(func=m.update_profile)

environment_update = subparsers.add_parser('environment')
environment_update.add_argument('environment_name')
environment_update.set_defaults(func=m.update_environment)

options = parser.parse_args()
options.func(options)

print(m.method_calls)

Python: muchos fun :(

python lusis.py -h

usage: lusis.py [-h] {query,add,update} ...

positional arguments:
  {query,add,update}

optional arguments:
  -h, --help          show this help message and exit

python lusis.py query -h

usage: lusis.py query [-h] {host,profile,environment} ...
 
positional arguments:
   {host,profile,environment}

optional arguments:
   -h, --help            show this help message and exit

John E. ("lusis") Vincent lusis/python argparse subcommand subparsers.py

Python: subcommands with `argparse`

cmd query host host_name
cmd query profile profile_name
cmd query environment environment_name
cmd add host host_name
cmd add profile profile_name
cmd add environment environment_name
cmd update host host_name

Observation: Imperative

Sandwich xkcd

Python: The CLI Renaissance

plac
click

Python: `plac`

import plac

@plac.flg('list_')  # avoid clash with builtin
@plac.flg('yield_')  # avoid clash with keyword
@plac.opt('sys_')  # avoid clash with a very common name
def main(list_, yield_=False, sys_=100):
    print(list_)
    print(yield_)
    print(sys_)

if __name__ == '__main__':
    plac.call(main)

bash$ python example13.py --help 
usage: example13.py [-h] [-l] [-y] [-s 100]

optional arguments:
  -h, --help         show this help message and exit
  -l, --list
  -y, --yield        [False]
  -s 100, --sys 100  [100]

plac Python package that can generate command line parameters from function signatures

Python: `click`

import click

@click.command()
@click.option('--count', default=1, help='Number of greetings.')
@click.option('--name', prompt='Your name',
              help='The person to greet.')
def hello(count, name):
    """Simple program that greets NAME for a total of COUNT times."""
    for x in range(count):
        click.echo(f"Hello {name}!")

if __name__ == '__main__':
    hello()

bash$ python hello.py --help
Usage: hello.py [OPTIONS]

  Simple program that greets NAME for a total of COUNT times.

Options:
  --count INTEGER  Number of greetings.
  --name TEXT      The person to greet.
  --help           Show this message and exit.

Python package for creating beautiful command line interfaces in a composable way with as little code as necessary.
It's the "Command Line Interface Creation Kit".

C/C++

c++-survey.png

A survey of argument parsing libraries in C/C++ attractivechaos (2018-08-31)

C: `argparse`

#include 
#include 
#include 
#include "argparse.h"

#define ARRAY_SIZE(x) (sizeof(x) / sizeof(x[0]))

static const char *const usages[] = {
    "subcommands [options] [cmd] [args]",
    NULL,
};

struct cmd_struct {
    const char *cmd;
    int (*fn) (int, const char **);
};

int
cmd_foo(int argc, const char **argv)
{
    printf("executing subcommand foo\n");
    printf("argc: %d\n", argc);
    for (int i = 0; i < argc; i++) {
        printf("argv[%d]: %s\n", i, *(argv + i));
    }
    int force = 0;
    int test = 0;
    const char *path = NULL;
    struct argparse_option options[] = {
        OPT_HELP(),
        OPT_BOOLEAN('f', "force", &force, "force to do", NULL, 0, 0),
        OPT_BOOLEAN('t', "test", &test, "test only", NULL, 0, 0),
        OPT_STRING('p', "path", &path, "path to read", NULL, 0, 0),
        OPT_END(),
    };
    struct argparse argparse;
    argparse_init(&argparse, options, usages, 0);
    argc = argparse_parse(&argparse, argc, argv);
    printf("after argparse_parse:\n");
    printf("argc: %d\n", argc);
    for (int i = 0; i < argc; i++) {
        printf("argv[%d]: %s\n", i, *(argv + i));
    }
    return 0;
}

int
cmd_bar(int argc, const char **argv)
{
    printf("executing subcommand bar\n");
    for (int i = 0; i < argc; i++) {
        printf("argv[%d]: %s\n", i, *(argv + i));
    }
    return 0;
}

static struct cmd_struct commands[] = {
    {"foo", cmd_foo},
    {"bar", cmd_bar},
};

int
main(int argc, const char **argv)
{
    struct argparse argparse;
    struct argparse_option options[] = {
        OPT_HELP(),
        OPT_END(),
    };
    argparse_init(&argparse, options, usages, ARGPARSE_STOP_AT_NON_OPTION);
    argc = argparse_parse(&argparse, argc, argv);
    if (argc < 1) {
        argparse_usage(&argparse);
        return -1;
    }

    /* Try to run command with args provided. */
    struct cmd_struct *cmd = NULL;
    for (int i = 0; i < ARRAY_SIZE(commands); i++) {
        if (!strcmp(commands[i].cmd, argv[0])) {
            cmd = &commands[i];
        }
    }
    if (cmd) {
        return cmd->fn(argc, argv);
    }
    return 0;
}

argparse A command line arguments parsing library in C (compatible with C++)

C++: `ArgumentParser`

#include 

#include 
#include 
#include       // For printing SeqAn Strings.

#include 

using namespace seqan;

int main(int argc, char const ** argv)
{
    // Initialize ArgumentParser.
    ArgumentParser parser("arg_parse_demo");
    setCategory(parser, "Demo");
    setShortDescription(parser, "Just a demo of the new ArgumentParser!");
    setVersion(parser, "0.1");
    setDate(parser, "Mar 2012");

    // Add use and description lines.
    addUsageLine(parser, "[\\fIOPTIONS\\fP] \\fIIN\\fP \\fIOUT\\fP ");

    addDescription(
        parser,
        "This is just a little demo to show what ArgumentParser is "
        "able to do.  \\fIIN\\fP is a multi-FASTA input file.  \\fIOUT\\fP is a "
        "txt output file.");

    // Add positional arguments and set their valid file types.
    addArgument(parser, ArgParseArgument(ArgParseArgument::INPUT_FILE, "IN"));
    addArgument(parser, ArgParseArgument(ArgParseArgument::OUTPUT_FILE, "OUT"));
    setValidValues(parser, 0, "FASTA fa");
    setValidValues(parser, 1, "txt");

    // Add a section with some options.
    addSection(parser, "Important Tool Parameters");
    addOption(parser, ArgParseOption("", "id", "Sequence identity between [0.0:1.0]",
                                     ArgParseArgument::DOUBLE, "ID"));
    setRequired(parser, "id", true);
    setMinValue(parser, "id", "0.0");
    setMaxValue(parser, "id", "1.0");

    // Adding a verbose and a hidden option.
    addSection(parser, "Miscellaneous");
    addOption(parser, ArgParseOption("v", "verbose", "Turn on verbose output."));
    addOption(parser, ArgParseOption("H", "hidden", "Super mysterious flag that will not be shown in "
                                                    "the help screen or man page."));
    hideOption(parser, "H");

    // Add a Reference section.
    addTextSection(parser, "References");
    addText(parser, "http://www.seqan.de");

    // Parse the arguments.
    ArgumentParser::ParseResult res = parse(parser, argc, argv);
    // Return if there was an error or a built-in command was triggered (e.g. help).
    if (res != ArgumentParser::PARSE_OK)
        return res == ArgumentParser::PARSE_ERROR;  // 1 on errors, 0 otherwise

    // Extract and print the options.
    bool verbose = false;
    getOptionValue(verbose, parser, "verbose");
    std::cout < "Verbose:     " < (verbose ? "on" : "off") < std::endl;

    double identity = -1.0;
    getOptionValue(identity, parser, "id");
    std::cout < "Identity:    " < identity < std::endl;

    CharString inputFile, outputFile;
    getArgumentValue(inputFile, parser, 0);
    getArgumentValue(outputFile, parser, 1);

    std::cout < "Input-File:  " < inputFile < std::endl;
    std::cout < "Output-File: " < outputFile < std::endl;

    return 0;
}

ArgumentParser Parse the command line

Java: `picocli`

/**
 * ASCII Art: Basic Picocli based sample application
 * Explanation: Picocli quick guide
 * Source Code: GitHub 
 * @author Andreas Deininger
 */
import picocli.CommandLine;
import picocli.CommandLine.Command;
import picocli.CommandLine.Option;
import picocli.CommandLine.Parameters;

import java.awt.Font;
import java.awt.Graphics;
import java.awt.Graphics2D;
import java.awt.RenderingHints;
import java.awt.image.BufferedImage;

@Command(name = "ASCIIArt", version = "ASCIIArt 1.0", mixinStandardHelpOptions = true) // |1|
public class ASCIIArt implements Runnable { // |2|

    @Option(names = { "-s", "--font-size" }, description = "Font size") // |3|
    int fontSize = 14;

    @Parameters(paramLabel = "", defaultValue = "Hello, picocli",  // |4|
               description = "Words to be translated into ASCII art.")
    private String[] words = { "Hello,", "picocli" }; // |5|

    @Override
    public void run() { // |6|
        // https://stackoverflow.com/questions/7098972/ascii-art-java
        BufferedImage image = new BufferedImage(144, 32, BufferedImage.TYPE_INT_RGB);
        Graphics graphics = image.getGraphics();
        graphics.setFont(new Font("Dialog", Font.PLAIN, fontSize));
        Graphics2D graphics2D = (Graphics2D) graphics;
        graphics2D.setRenderingHint(RenderingHints.KEY_TEXT_ANTIALIASING,
                RenderingHints.VALUE_TEXT_ANTIALIAS_ON);
        graphics2D.drawString(String.join(" ", words), 6, 24);

        for (int y = 0; y < 32; y++) {
            StringBuilder builder = new StringBuilder();
            for (int x = 0; x < 144; x++)
                builder.append(image.getRGB(x, y) == -16777216 ? " " : image.getRGB(x, y) == -1 ? "#" : "*");
            if (builder.toString().trim().isEmpty()) continue;
            System.out.println(builder);
        }
    }

    public static void main(String[] args) {
        int exitCode = new CommandLine(new ASCIIArt()).execute(args); // |7|
        System.exit(exitCode); // |8|
    }
}

picocli—a mighty tiny command line interface

Python-le-Nouveau

Subcommands & options

cmd hdfs stop
cmd hdfs start
cmd spark stop cluster --instance=instance-2 --verbose
cmd spark start cluster --instance=instance-1 --quietly

Python: pure data

view = {
}

Python: pure data

cmd spark stop cluster --instance=instance-2 --verbose

view = {
    'spark|stop|cluster': {
    },
}

Python: pure data

cmd spark stop cluster --instance=instance-2 --verbose

view = {
    'spark|stop|cluster': {
        'callback':  spark_stop_cluster,
    },
}

Python: pure data

cmd spark stop cluster --instance=instance-2 --verbose

view = {
    'spark|stop|cluster': {
        'callback':  spark_stop_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
}

Observation: MVC

Model-view-controller Wikipedia

Python: View (pure data)

cmd spark start cluster --instance=instance-2 --verbose
cmd spark stop cluster --instance=instance-2 --verbose

view = {
    'spark|start|cluster': {
        'callback':  spark_start_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
    'spark|stop|cluster': {
        'callback':  spark_stop_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
}

Python: View (pure data)

view = {
    'spark|start|cluster': {
        'callback':  spark_start_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
    'spark|stop|cluster': {
        'callback':  spark_stop_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
}

Python: View (pure data)

view = {
    'spark|start|cluster': {
        'callback':  spark_start_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
    'spark|stop|cluster': {
        'callback':  spark_stop_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
    'hdfs|start': {
        'callback':  hdfs_start,
        'help':     'Start Hadoop',
    },
    'hdfs|stop': {
        'callback':  hdfs_stop,
        'help':     'Stop Hadoop',
    },
}

Python: complete program

import sys
from parsearg import ParseArg
from spark import (
    spark_start_cluster,
    spark_stop_cluster,
)
from hdfs import (
    hdfs_start,
    hdfs_stop,
)

view = {
    'spark|start': {
        'callback':  spark_start_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
    'spark|stop': {
        'callback':  spark_stop_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
    'hdfs|start': {
        'callback':  hdfs_start,
        'help':     'Start Hadoop',
    },
    'hdfs|stop': {
        'callback':  hdfs_stop,
        'help':     'Stop Hadoop',
    },
}

def main(args: list)-> None:
    parser = ParseArg(view)
    ns     = parser.parse_args(args)
    result = ns.callback(ns)

if __name__ == "__main__":
    args = sys.argv[1:] if len(sys.argv) > 1 else []
    main(' '.join(args))

Subcommands & options

cmd hdfs stop
cmd hdfs start
cmd spark stop cluster --instance=instance-2 --verbose
cmd spark start cluster --instance=instance-1 --quietly

Subcommands: reordered

cmd cluster stop hdfs
cmd cluster start hdfs 
cmd cluster stop spark --instance=instance-2 --verbose
cmd cluster start spark --instance=instance-1 --quietly

Subcommands: reordered

cmd spark stop cluster --instance=instance-2 --verbose
cmd cluster stop spark --instance=instance-2 --verbose

Python: View—original order

cmd hdfs stop
cmd hdfs start
cmd spark stop cluster --instance=instance-2 --verbose
cmd spark start cluster --instance=instance-1 --quietly

view = {
    'spark|start|cluster': {
        'callback':  spark_start_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
    'spark|stop|cluster': {
        'callback':  spark_stop_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
    'hdfs|start': {
        'callback':  hdfs_start,
        'help':     'Start Hadoop',
    },
    'hdfs|stop': {
        'callback':  hdfs_stop,
        'help':     'Stop Hadoop',
    },
}

Python: View—reordered subcommands

cmd cluster stop hdfs
cmd cluster start hdfs 
cmd cluster stop spark --instance=instance-2 --verbose
cmd cluster start spark --instance=instance-1 --quietly

view = {
    'cluster|start|spark': {
        'callback':  spark_start_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
    'cluster|stop|spark': {
        'callback':  spark_stop_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
    'cluster|start|hdfs': {
        'callback':  hdfs_start,
        'help':     'Start Hadoop',
    },
    'cluster|stop|hdfs': {
        'callback':  hdfs_stop,
        'help':     'Stop Hadoop',
    },
}

R

Subcommands & options

cmd hdfs stop
cmd hdfs start
cmd spark stop cluster --instance=instance-2 --verbose
cmd spark start cluster --instance=instance-1 --quietly

R: View (pure data)

view = list(
)

R: View (pure data)

view = list(
    `spark|start|cluster` = list()
)

R: View (pure data)

view = list(
    `spark|start|cluster` = list(
        `callback` = spark_start_cluster
    )
)

R: View (pure data)

view = list(
    `spark|start|cluster` = list(
        `callback`    = spark_start_cluster,
        `--instance`  = list(
            `help`    =  'Spark cluster instance',
            `default` =  'instance-1',
            `choices` =   c('instance-1', 'instance-2')
        )
    )
)

R: View (pure data)

view = list(
    `spark|start|cluster` = list(
        `callback`    = spark_start_cluster,
        `--instance`  = list(
            `help`    =  'Spark cluster instance',
            `default` =  'instance-1',
            `choices` =   c('instance-1', 'instance-2')
        )
    ),
    `spark|stop|cluster` = list(
        `callback`    = spark_stop_cluster,
        `--instance`  = list(
            `help`    =  'Spark cluster instance',
            `default` =  'instance-1',
            `choices` =   c('instance-1', 'instance-2')
        )
    )
)

R: View (pure data)

view = list(
    `spark|start|cluster` = list(
        `callback`    = spark_start_cluster,
        `--instance`  = list(
            `help`    =  'Spark cluster instance',
            `default` =  'instance-1',
            `choices` =   c('instance-1', 'instance-2')
        )
    ),
    `spark|stop|cluster` = list(
        `callback`    = spark_stop_cluster,
        `--instance`  = list(
            `help`    =  'Spark cluster instance',
            `default` =  'instance-1',
            `choices` =   c('instance-1', 'instance-2')
        )
    ),
    `hdfs|start` = list(
        `callback`    = hdfs_start,
        `--instance`  = list(
            `help`    =  'Start Hadoop'
        )
    ),
    `hdfs|stop` = list(
        `callback`    = hdfs_stop,
        `--instance`  = list(
            `help`    =  'Stop Hadoop'
        )
    )
)

R: complete program

 #!/usr/bin/Rscript --no-init-file

suppressPackageStartupMessages(library(parsearg))

`spark_start_cluster`<- function(args) 
    cat(sprintf("starting Spark cluster instance=%s", args.instance)),
`spark_stop_cluster`<- function(args) 
    cat(sprintf("stopping Spark cluster instance=%s", args.instance)),
`hdfs_start`<- function(args) 
    cat(sprintf("starting HDFS cluster instance=%s", args.instance)),
`hdfs_stop`<- function(args) 
    cat(sprintf("stopping HDFS cluster instance=%s", args.instance)),

view = list(
    `spark|start|cluster` = list(
        `callback`    = spark_start_cluster,
        `--instance`  = list(
            `help`    =  'Spark cluster instance',
            `default` =  'instance-1',
            `choices` =   c('instance-1', 'instance-2')
        )
    ),
    `spark|stop|cluster` = list(
        `callback`    = spark_stop_cluster,
        `--instance`  = list(
            `help`    =  'Spark cluster instance',
            `default` =  'instance-1',
            `choices` =   c('instance-1', 'instance-2')
        )
    ),
    `hdfs|start` = list(
        `callback`    = hdfs_start,
        `--instance`  = list(
            `help`    =  'Start Hadoop'
        )
    ),
    `hdfs|stop` = list(
        `callback`    = hdfs_stop,
        `--instance`  = list(
            `help`    =  'Stop Hadoop'
        )
    )
)

`main`<- function(args=commandArgs(trailingOnly=TRUE)) {
    parser<- ParseArg(view)
    ns<-     parser$parse_args(args)
    result<- ns$callback(ns)
}
main()

R: complete program (alt)

 #!/usr/bin/Rscript --no-init-file

suppressPackageStartupMessages(library(parsearg))

`spark_start_cluster`<- function(args) 
    cat(sprintf("starting Spark cluster instance=%s", args.instance)),
`spark_stop_cluster`<- function(args) 
    cat(sprintf("stopping Spark cluster instance=%s", args.instance)),
`hdfs_start`<- function(args) 
    cat(sprintf("starting HDFS cluster instance=%s", args.instance)),
`hdfs_stop`<- function(args) 
    cat(sprintf("stopping HDFS cluster instance=%s", args.instance)),

view = list(
    `spark|start|cluster` = list(
        `callback`    = spark_start_cluster,
        `--instance`  = list(
            `help`    =  'Spark cluster instance',
            `default` =  'instance-1',
            `choices` =   c('instance-1', 'instance-2')
        )
    ),
    `spark|stop|cluster` = list(
        `callback`    = spark_stop_cluster,
        `--instance`  = list(
            `help`    =  'Spark cluster instance',
            `default` =  'instance-1',
            `choices` =   c('instance-1', 'instance-2')
        )
    ),
    `hdfs|start` = list(
        `callback`    = hdfs_start,
        `--instance`  = list(
            `help`    =  'Start Hadoop'
        )
    ),
    `hdfs|stop` = list(
        `callback`    = hdfs_stop,
        `--instance`  = list(
            `help`    =  'Stop Hadoop'
        )
    )
)

args<-   commandArgs(trailingOnly=TRUE)
parser<- ParseArg(view)
ns<-     parser$parse_args(args)
result<- ns$callback(ns)

Observations

DSL:
- positional arguments
- options
- subcommands
Recursion: subcommands
Tree: hierarchy + terminating case
Declarative not Imperative: pure data
MVC: separation of concerns

University of CLI

UCLI @ DSL

`argparse`: Python

Part of Python's standard library

`argparse`: R

R version calls Python

`parsearg`: Python, R

Canonical example


#!/usr/bin/Rscript --no-init-file

library(argparse)

`square`<- function(args) {
    parser<- ArgumentParser()

    parser$add_argument("number", 
        type="integer",
        help="display the square of a given number"
    )

    parser$add_argument("-v", "--verbosity", 
        action="count",
        default=0,
        help="increase output verbosity"
    )

    args<-   parser$parse_args(args)
    answer<- args$number^2

    if (args$verbosity > 2) {
        o<- capture.output(example("^"))

        cat(sprintf("[verbosity level == %d]:\n", args$verbosity))
        cat(sprintf(
            "the square of %d equals %d, using the in-built arithmetic operator '^':\n",
            args$number, answer
        ))
        cat(sprintf("%s\n", o))
    }
    else if (args$verbosity == 2) {
        cat(sprintf("[verbosity level == %d]:\n", args$verbosity))
        cat(sprintf("the square of %d equals %d\n", args$number, answer))
    }
    else if (args$verbosity == 1) {
        cat(sprintf("the square of %d is %d\n", args$number, answer))
    }
    else {
        cat(sprintf("answer = %d\n", answer))
    }
}

square(args=commandArgs(trailingOnly=TRUE))


bash$ R --no-init-file

R version 4.2.0 (2022-04-22) -- "Vigorous Calisthenics"
Copyright (C) 2022 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> source('square')
> square('4') 
> square('4 -v') 
> square('4 -vv') 
> square('4 -vvvv') 


bash$ square 4
bash$ square 4 -v
bash$ square 4 -vv
bash$ square 4 -vvvv


bash$ ./square 2
answer = 4

bash$ ./square 2 -v
the square of 2 is 4

bash$ ./square 2 -vv
[verbosity level == 2]: 
the square of 2 equals 4

bash$ ./square 2 -vvv
[verbosity level == 3]:
the square of 2 equals 4, using the in-built arithmetic operator '^':

 ^> x <- -1:12
 
 ^> x + 1
  [1]  0  1  2  3  4  5  6  7  8  9 10 11 12 13
 
 ^> 2 * x + 3
  [1]  1  3  5  7  9 11 13 15 17 19 21 23 25 27
 
 ^> x %% 2 #-- is periodic
  [1] 1 0 1 0 1 0 1 0 1 0 1 0 1 0
 
 ^> x %/% 5
  [1] -1  0  0  0  0  0  1  1  1  1  1  2  2  2
 
 ^> x %% Inf # now is defined by limit (gave NaN in earlier versions of R)
  [1] Inf   0   1   2   3   4   5   6   7   8   9  10  11  12

Command-line arguments

DEFINITION. A command line comprises command-line arguments:

the program name (first); and,
[other command-line arguments].

The essence of an argument parser

firstly, associates a command-line argument with a parser argument

secondly, associates the parser argument with a variable name

The essence of an argument parser

command-line argument -> parser argument -> variable name

The essence of an argument parser

command-line "event" -> variable name

The essence of an argument parser

command-line "event" -> handle the event

The essence of an argument parser

event handling

Command-line arguments: Unnamed


bash$ square 3.141593
             ^^^^^^^^  # UNNAMED command-line argument:
                       # it is not named on the command line;
                       # the "event" is determined by its position

Command-line arguments: Unnamed


#!/usr/bin/Rscript --no-init-file

library(argparse)

`square`<- function(args) {
    parser<- ArgumentParser()

    parser$add_argument("number", 
        type="integer",
        help="display the square of a given number"
    )

    parser$add_argument("-v", "--verbosity", 
        action="count",
        default=0,
        help="increase output verbosity"
    )

    args<-   parser$parse_args(args)
    answer<- args$number^2

    if (args$verbosity > 2) {
        o<- capture.output(example("^"))

        cat(sprintf("[verbosity level == %d]:\n", args$verbosity))
        cat(sprintf(
            "the square of %d equals %d, using the in-built arithmetic operator '^':\n",
            args$number, answer
        ))
        cat(sprintf("%s\n", o))
    }
    else if (args$verbosity == 2) {
        cat(sprintf("[verbosity level == %d]:\n", args$verbosity))
        cat(sprintf("the square of %d equals %d\n", args$number, answer))
    }
    else if (args$verbosity == 1) {
        cat(sprintf("the square of %d is %d\n", args$number, answer))
    }
    else {
        cat(sprintf("answer = %d\n", answer))
    }
}

square(args=commandArgs(trailingOnly=TRUE))

Command-line arguments: Named


bash$ square -n 3.141593
             ^^           # SHORT ARG

bash$ square -n=3.141593
             ^^           # SHORT ARG

bash$ square -number 3.141593
             ^^^^^^^      # FLAG

bash$ square --number=3.141593
             ^^^^^^^^     # FLAG

Command-line arguments: Named


#!/usr/bin/Rscript --no-init-file

library(argparse)

`square`<- function(args) {
    parser<- ArgumentParser()

    parser$add_argument("-n", "--number", 
        type="integer",
        help="display the square of a given number"
    )

    parser$add_argument("-v", "--verbosity", 
        action="count",
        default=0,
        help="increase output verbosity"
    )

    args<-   parser$parse_args(args)
    answer<- args$number^2

    if (args$verbosity > 2) {
        o<- capture.output(example("^"))

        cat(sprintf("[verbosity level == %d]:\n", args$verbosity))
        cat(sprintf(
            "the square of %d equals %d, using the in-built arithmetic operator '^':\n",
            args$number, answer
        ))
        cat(sprintf("%s\n", o))
    }
    else if (args$verbosity == 2) {
        cat(sprintf("[verbosity level == %d]:\n", args$verbosity))
        cat(sprintf("the square of %d equals %d\n", args$number, answer))
    }
    else if (args$verbosity == 1) {
        cat(sprintf("the square of %d is %d\n", args$number, answer))
    }
    else {
        cat(sprintf("answer = %d\n", answer))
    }
}

square(args=commandArgs(trailingOnly=TRUE))

Command-line arguments: Named


#!/usr/bin/Rscript --no-init-file

library(argparse)

`square`<- function(args) {
    parser<- ArgumentParser()

    parser$add_argument("-n", "--number", 
        type="integer",
        help="display the square of a given number",
        dest="number_to_square"
    )

    parser$add_argument("-v", "--verbosity", 
        action="count",
        default=0,
        help="increase output verbosity"
    )

    args<-   parser$parse_args(args)
    answer<- args$number_to_square^2

    if (args$verbosity > 2) {
        o<- capture.output(example("^"))

        cat(sprintf("[verbosity level == %d]:\n", args$verbosity))
        cat(sprintf(
            "the square of %d equals %d, using the in-built arithmetic operator '^':\n",
            args$number_to_square, answer
        ))
        cat(sprintf("%s\n", o))
    }
    else if (args$verbosity == 2) {
        cat(sprintf("[verbosity level == %d]:\n", args$verbosity))
        cat(sprintf("the square of %d equals %d\n", args$number_to_square, answer))
    }
    else if (args$verbosity == 1) {
        cat(sprintf("the square of %d is %d\n", args$number_to_square, answer))
    }
    else {
        cat(sprintf("answer = %d\n", answer))
    }
}

square(args=commandArgs(trailingOnly=TRUE))

Positional and non-positional arguments


ArgumentParser.add_argument(
    name or flags...
    [, action]
    [, nargs]
    [, const]
    [, default]
    [, type]
    [, choices]
    [, required]
    [, help]
    [, metavar]
    [, dest]
)

Define how a single command-line argument should be parsed. Each parameter
has its own more detailed description below, but in short they are:


name or flags - Either a name or a list of option strings, 
                e.g. foo or -f, --foo.
action        - The basic type of action to be taken when this argument is 
                encountered at the command line.
nargs         - The number of command-line arguments that should be consumed.
const         - A constant value required by some action and nargs selections.
default       - The value produced if the argument is absent from the command line.
type          - The type to which the command-line argument should be converted.
choices       - A container of the allowable values for the argument.
required      - Whether or not the command-line option may be omitted 
                (optionals only).
help          - A brief description of what the argument does.
metavar       - A name for the argument in usage messages.
dest          - The name of the attribute to be added to the object returned 
                by parse_args().

Python docs: ArgumentParser.add_argument

Positional and non-positional arguments

The method add_argument must know whether a positional argument or an optional argument is being added.
What distinguishes a "positional argument" from an "optional argument"?
The term ``optional argument'' is therefore unfortunate if the the argument is a required argument: In this case, ``optional'' really means that the argument is a required non-positional argument.
The terminology is therefore as unfortunate as it is confusing. Optional arguments would perhaps better be referred to as non-positional arguments.
A positional argument is a name like foo. An optional argument is a name prefixed by a short arg (something starting with - ) or a flag (something starting with -- ). We can summarize thus:
positional argument
- occupies a specific position on the command line
- required argument ipso facto
- has a parser-argument name like foo
non-positional argument (a.k.a ``optional'')
- does not occupy a specific position on the command line
- not required by default
- all optional arguments are prefixed by either - or --
- a short arg like -f or a flag like --foo

positional argument (unnamed):

required: by default a positional argument ("name") must occupy a particular position on the command line

non-positional argument (named):

not required: by default a non-positional ("optional") argument can be in position on the command line

Positional and non-positional arguments

The method add_argument must know whether a positional argument or an optional argument is being added.
What distinguishes a "positional argument" from an "optional argument"?
The term ``optional argument'' is therefore unfortunate if the the argument is a required argument: In this case, ``optional'' really means that the argument is a required non-positional argument.
The terminology is therefore as unfortunate as it is confusing. Optional arguments would perhaps better be referred to as non-positional arguments.
A positional argument is a name like foo. An optional argument is a name prefixed by a short arg (something starting with - ) or a flag (something starting with -- ). We can summarize thus:
positional argument
- occupies a specific position on the command line
- required argument ipso facto
- has a parser-argument name like foo
non-positional argument (a.k.a ``optional'')
- does not occupy a specific position on the command line
- not required by default
- all optional arguments are prefixed by either - or --
- a short arg like -f or a flag like --foo

Unnamed and named arguments

The method add_argument must know whether a positional argument or an optional argument is being added.
What distinguishes a "positional argument" from an "optional argument"?
The term ``optional argument'' is therefore unfortunate if the the argument is a required argument: In this case, ``optional'' really means that the argument is a required non-positional argument.
The terminology is therefore as unfortunate as it is confusing. Optional arguments would perhaps better be referred to as non-positional arguments.
A positional argument is a name like foo. An optional argument is a name prefixed by a short arg (something starting with - ) or a flag (something starting with -- ). We can summarize thus:
positional argument
- occupies a specific position on the command line
- required argument ipso facto
- has a parser-argument name like foo
non-positional argument (a.k.a ``optional'')
- does not occupy a specific position on the command line
- not required by default
- all optional arguments are prefixed by either - or --
- a short arg like -f or a flag like --foo

Arguments to `add_arguments`


ArgumentParser.add_argument(
    name or flags...
    [, action]
    [, nargs]
    [, const]
    [, default]
    [, type]
    [, choices]
    [, required]
    [, help]
    [, metavar]
    [, dest]
)

Define how a single command-line argument should be parsed. Each parameter
has its own more detailed description below, but in short they are:


name or flags - Either a name or a list of option strings, 
                e.g. foo or -f, --foo.
action        - The basic type of action to be taken when this argument is 
                encountered at the command line.
nargs         - The number of command-line arguments that should be consumed.
const         - A constant value required by some action and nargs selections.
default       - The value produced if the argument is absent from the command line.
type          - The type to which the command-line argument should be converted.
choices       - A container of the allowable values for the argument.
required      - Whether or not the command-line option may be omitted 
                (optionals only).
help          - A brief description of what the argument does.
metavar       - A name for the argument in usage messages.
dest          - The name of the attribute to be added to the object returned 
                by parse_args().


#!/usr/bin/Rscript --no-init-file

library(argparse)

`square`<- function(args) {
    parser<- ArgumentParser()

    parser$add_argument("-n", "--number", 
        type="integer",
        help="display the square of a given number",
        dest="number_to_square"
    )

    parser$add_argument("-v", "--verbosity", 
        action="count",
        default=0,
        help="increase output verbosity"
    )

    args<-   parser$parse_args(args)
    answer<- args$number_to_square^2

    if (args$verbosity > 2) {
        o<- capture.output(example("^"))

        cat(sprintf("[verbosity level == %d]:\n", args$verbosity))
        cat(sprintf(
            "the square of %d equals %d, using the in-built arithmetic operator '^':\n",
            args$number_to_square, answer
        ))
        cat(sprintf("%s\n", o))
    }
    else if (args$verbosity == 2) {
        cat(sprintf("[verbosity level == %d]:\n", args$verbosity))
        cat(sprintf("the square of %d equals %d\n", args$number_to_square, answer))
    }
    else if (args$verbosity == 1) {
        cat(sprintf("the square of %d is %d\n", args$number_to_square, answer))
    }
    else {
        cat(sprintf("answer = %d\n", answer))
    }
}

square(args=commandArgs(trailingOnly=TRUE))

Python docs: ArgumentParser.add_argument

DSL: additional arguments


(
    name or flags...
    [, callback]
    [, local]
    [, action]
    [, nargs]
    [, const]
    [, default]
    [, type]
    [, choices]
    [, required]
    [, help]
    [, metavar]
    [, dest]
)

parsearg

DSL: keys

view = list(
    `spark|start|cluster` = list(
        `callback`    = spark_start_cluster,
        `--instance`  = list(
            `help`    =  'Spark cluster instance',
            `default` =  'instance-1',
            `choices` =   c('instance-1', 'instance-2')
        )
    ),
    `spark|stop|cluster` = list(
        `callback`    = spark_stop_cluster,
        `--instance`  = list(
            `help`    =  'Spark cluster instance',
            `default` =  'instance-1',
            `choices` =   c('instance-1', 'instance-2')
        )
    ),
    `hdfs|start` = list(
        `callback`    = hdfs_start,
        `--instance`  = list(
            `help`    =  'Start Hadoop'
        )
    ),
    `hdfs|stop` = list(
        `callback`    = hdfs_stop,
        `--instance`  = list(
            `help`    =  'Stop Hadoop'
        )
    )
)

DSL: keys

keys form a flattened tree

start|spark|cluster
start -> spark -> cluster

abstractly

A|B|C
A -> B -> C

neat summary of nested hierarchy of subcommands

the magic of parsearg is in unflattening the flat tree into a tree of argparse parsers

DSL: unflattening

UCLI @ Recursion & Trees

Heterogeneous trees: recursive data structures

An heterogeneous tree is either:

a value and a set of child trees; or,
the empty tree.

Trees: Flat & Unflattened

cmd hdfs stop 
cmd hdfs start 
cmd spark stop cluster --instance=instance-2 --verbose
cmd spark start cluster --instance=instance-1 --quietly

view = {
    'spark|start|cluster': {
        'callback':  spark_start_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
    'spark|stop|cluster': {
        'callback':  spark_stop_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
    'hdfs|start': {
        'callback':  hdfs_start,
        'help':     'Start Hadoop',
    },
    'hdfs|stop': {
        'callback':  hdfs_stop,
        'help':     'Stop Hadoop',
    },
}

Trees: Flat & Unflattened

cmd cluster stop hdfs
cmd cluster start hdfs 
cmd cluster stop spark --instance=instance-2 --verbose
cmd cluster start spark --instance=instance-1 --quietly

view = {
    'cluster|start|spark': {
        'callback':  spark_start_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
    'cluster|stop|spark': {
        'callback':  spark_stop_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
    'cluster|start|hdfs': {
        'callback':  hdfs_start,
        'help':     'Start Hadoop',
    },
    'cluster|stop|hdfs': {
        'callback':  hdfs_stop,
        'help':     'Stop Hadoop',
    },
}

Trees: longhand

"""
       A
     / | \
    B  BB BBB
      /| \
     C CC CCC
"""

from parsearg.data_structures import Tree, Node, Key

tree = Tree('A', children=[
    Tree('B', []),
    Tree('BB', children=[
        Tree('C', []),
        Tree('CC', []),
        Tree('CCC', [])
    ]),
    Tree('BBB', []),
])

print(tree.show(quiet=True))


A
    B
    BB
        C
        CC
        CCC
    BBB

Trees: longhand

`Tree`<- function(value=NULL, children=NULL) {
    value<-    force(value)
    children<- force(children)
    if (is.null(children))
        children<- list()

    .Value<- function(name=NULL, key=NULL, d=NULL) {
        `check`<- function(x) {
            if (!is.null(x))
                assert(class(x)=="character")
        }
        check(name)
        check(key)
        if (!is.null(d)) {
            assert(
                class(d)=="list",
                !is.null(key)
            )
            payload=d[[key]]
        }

        list(
            name=name,
            key=key,
            payload=d
        )
    }
    .show<- function(level=0, indent = '\t', quiet=FALSE) {
        `.is_empty`<- function(tree) {
            is.null(tree$value) && length(tree$children) == 0
        }

        o<- list()
        `.show`<- function(tree, level, indent) {
            # depth-first search (DFS)
            if (.is_empty(tree)) return
            o<- c(o, sprintf("%s%s", strrep(indent, level), tree$value))
            for (child in tree$children) {
                .show(child, level=level+1, indent=indent)
            }
        }

        self<- Tree(value, children)
        .show(self, level, indent)
        o = paste0(paste0(o, collapse='\n'), "\n")

        if (!quiet) print(o)

        o
    }

    structure(
        list(
            `value`    = value,
            `children` = children,
            `show`     = .show,
            `Value`    = .Value
        ),
        class="Tree"
    )
}

Trees: longhand

Tree('A', children=list(
    Tree('B'),
    Tree('BB', children=list(
        Tree('C'),
        Tree('CC'),
        Tree('CCC')
    )),
    Tree('BBB')
))$show(quiet=TRUE, indent="    ") %>% cat


A
    B
    BB
        C
        CC
        CCC
    BBB

Trees: shorthand

list(
    'A',
    'A|B',
    'A|BB',
    'A|BB|C',
    'A|BB|CC',
    'A|BB|CCC',
    'A|BBB'
)

Trees: shorthand

Non-terminating nodes:

c(
    'A',
    'A|BB'
)

Terminating nodes (leaves):

c(
    'A|B',
    'A|BB|C',
    'A|BB|CC',
    'A|BB|CCC',
    'A|BBB'
)

Trees: longhand/shorthand

There is a duality here:

we need a shorthand in order for the tree's structure to be specified in a simple way (the "flattened" shorthand); and
we need to make use of the structure of the tree itself (the "unflattened" tree).

Trees: longhand/shorthand

parsearg transforms a list, representing the tree's nodes, into a Tree:

the names of the list are shorthand for the tree's nodes and the values of the list are the nodes' payloads;
we need the shorthand of the list for ease of specification;
and we need the Tree to avail of the data structure's properties.

UCLI @ Declarative

Data, not actions

Putting together a CLI with argparse alone is nothing if not an exercise in imperative programming, and this has negative consequences:

it obfuscates the intention of the CLI design;
it is prone to errors;
it discourages CLI design in the first instance;
it makes debugging a CLI design very difficult; and
it makes refactoring or re-configuring the CLI design overly burdensome.

A concrete abstract tree

cmd A BB CCC -c -v
cmd A B -c -v
cmd A B -c --verbose

view = list(
    `A` = list(
        `callback` =     make_callback('A'),
        `-c` =           list(`help` = 'A [optional pi]', `action` = 'store_const', `const` = 3.141593),
        `-v|--verbose` = list(`help` = 'A verbosity', `action` = 'store_true')
    ),
    `A|B` = list(
        `callback` =     make_callback('A_B'),
        `-c` =           list(`help` = 'A B [optional pi]', `action` = 'store_const', `const` = 3.141593),
        `-v|--verbose` = list(`help` = 'A B verbosity', `action` = 'store_true')
    ),
    `A|BB` = list(
        `callback` =     make_callback('A_BB'),
        `-c` =           list(`help` = 'A BB [optional pi]', `action` = 'store_const', `const` = 3.141593),
        `-v|--verbose` = list(`help` = 'A BB erbosity', `action` = 'store_true')
    ),
    `A|BB|C` = list(
        `callback` =     make_callback('A_BB_C'),
        `-c` =           list(`help` = 'A BB C [optional pi]', `action` = 'store_const', `const` = 3.141593),
        `-v|--verbose` = list(`help` = 'A BB C verbosity', `action` = 'store_true')
    ),
    `A|BB|CC` = list(
        `callback` =     make_callback('A_BB_CC'),
        `-c` =           list(`help` = 'A BB CC [optional pi]', `action` = 'store_const', `const` = 3.141593),
        `-v|--verbose` = list(`help` = 'A BB CC verbosity', `action` = 'store_true')
    ),
    `A|BB|CCC` = list(
        `callback` =     make_callback('A_BB_CCC'),
        `-c` =           list(`help` = 'A BB CCC [optional pi]', `action` = 'store_const', `const` = 3.141593),
        `-v|--verbose` = list(`help` = 'A BB CCC verbosity', `action` = 'store_true')
    ),
    `A|BBB` = list(
        `callback` =     make_callback('A_BBB'),
        `-c` =           list(`help` = 'A BBB [optional pi]', `action` = 'store_const', `const` = 3.141593),
        `-v|--verbose` = list(`help` = 'A BBB verbosity', `action` = 'store_true')
    )
)

UCLI @ MVC

Separation of concerns

argparse has everything we need in terms of functionality; it's just clunky to use

Separation of concerns

parsearg is a layer over argparse that exposes argparse's functionality via a list:

the list is the View component of the Model-View-Controller ("MVC")
the list embeds callbacks from the Controller component, achieving a clean separation of duties;
by separating the View component into a dict, the CLI design can be expressed in a declarative way.

Separation of concerns

parsearg manifests the intention of the CLI design without having to specify how that design is implemented in terms of argparse's parsers and subparsers (parsearg does that for you)

`parsearg`

How does it work?

Helper functions: `Key`

# 1. take key
key = 'A|B|C'

Helper functions: `Key`

# 2. split string
sep = '|'
print( Key.split(key) )

 ['A', '|', 'B', '|', 'C']

Helper functions: `Key`

# 3. unflatten the split key (i.e. create a nested list)
unflattened = Key.unflatten(Key.split(key))
print(unflattened)


['A', ['B', ['C']]]

Helper functions: `Node`

# 4. create a Node from the unflattened list
node = Node.from_nested_list(unflattened)
print(node)


('A', ['B', ['C']])

Helper functions: `Node`

# 5. the node can now be shifted until the empty node is reached
print(node < 0)
print(node < 1)
print(node < 2)
print(node < 3)

('A', ['B', ['C']])
('B', ['C'])
('C', [])
(None, [])

Helper functions: `Key` to `Node`

print( key )
print( Key.to_node(key) )


A|B|C
('A', ['B', ['C']])

Helper functions: What does a `Node` do?

key = 'A|B|C'
n = Key.to_node(key) 
i = 0
while not n.is_empty():
    i += 1
    n = node < i
    
print(i)
print(node < i)

3
(None, [])

Helper functions: Track a payload

print( Key(key).key )


A|B|C

Helper functions: Track a payload

payload = 3.141593
d = {key: payload}
print( Key(key, d=d).d )

{'A|BB|C': 3.141593}

Helper functions: Track a payload

print( Key(key).value )
print( type(Key(key).value) )

('A', ['BB', ['C']])

Helper functions: Track a payload

payload = 3.141593
d = {key: payload}
print( Key(key, d).payload )

3.141593

Helper functions: Shifting a `Key`

print( Key(key) )
# same thing:
print( Key(key) < 0 )
# real shifts:
print( Key(key) < 1 )
print( Key(key) < 2 )
print( Key(key) < 3 )
# same thing:
print( Key(key) < 4 )

'A|BB|C': ('A', ['BB', ['C']])
'A|BB|C': ('A', ['BB', ['C']])
'A|BB|C': ('BB', ['C'])
'A|BB|C': ('C', [])
'A|BB|C': (None, [])
'A|BB|C': (None, [])

`parsearg`: The Guts

import sys
import argparse
from parsearg.data_structures import (
    is_empty,
    Tree,
    Key,
)
from parsearg.utils import (
    is_valid,
    print_list,
    is_list_of,
)


class ParseArg:
    def __init__(self, d, root_name='root'):
        assert isinstance(d, dict) and is_valid(d)

        self.d       = d
        self.tree    = ParseArg.to_tree(self.d, root_name=root_name)
        self.parser  = argparse.ArgumentParser(add_help=True)
        ParseArg.make_subparsers(self.tree, self.parser)

    def parse_args(self, args):
        args   = args.split() if isinstance(args, str) else args
        return self.parser.parse_args(args)

    @staticmethod
    def make_subparsers(tree, parser):
        def argparse_argument_name_or_flags(keys):
            keys = keys.split('|')
            keys = list(map(lambda x: x.strip(' \t\n\r'), keys))

            return keys

        def make_parser(node, subparsers):
            assert type(node.value) == Tree.Value
            parser = subparsers.add_parser(node.value.name)

            # print(f'node = {node.value}')
            if node.value.payload is not None:
                for key, value in node.value.payload.items():
                    if key == 'callback':
                        parser.set_defaults(
                            callback=value
                        )
                    else:
                        parser.add_argument(
                            *argparse_argument_name_or_flags(key), **value
                        )

            if len(node.children):
                subparsers = parser.add_subparsers()
                for child in node.children:
                    make_parser(child, subparsers)

        if not tree.is_empty():
            subparsers = parser.add_subparsers()
            for child in tree.children:
                make_parser(child, subparsers)

    @staticmethod
    def to_tree(d, root_name='root'):
        assert isinstance(d, dict)

        nodes = map(lambda key: Key(key), d.keys())
        nodes = list(filter(lambda x: not x.is_empty(), nodes))

        def _to_tree(x):
            if isinstance(x, list) and len(x)==1 and x[0].is_leaf():
                x = x[0]

                return Tree(
                    Tree.Value(name=x.value.head(), key=x.key, d=d)
                )

            else:
                name = set(map(lambda x: x.value.head(), x))
                assert len(name)==1
                name = name.pop()

                o       = []
                shifted = list(map(lambda x: x < 1, x))

                # optional arguments added to node itself
                tree = None
                empty   = list(filter(lambda x: x.is_empty(), shifted))
                assert len(empty) <= 1
                if len(empty):
                    x = empty[0]
                    tree = Tree(
                        Tree.Value(name = name, key=x.key, d=d)
                    )

                # arguments added to child nodes
                children    = list(filter(lambda x: not x.is_empty(), shifted))
                child_names = set(map(lambda x: x.value.head(), children))

                for child_name in child_names:
                    child = list(filter(lambda x: x.value.head() == child_name, children))
                    o    += [_to_tree(child)]

                if tree is not None:
                    tree.children = o 
                else:
                    tree = Tree(Tree.Value(name=name, key=None), children=o)

                return tree

        o = []
        for name in set(map(lambda x: x.value.head(), nodes)):
            node = list(filter(lambda x: x.value.head()==name, nodes))
            o += [_to_tree(node)]

        return Tree(root_name, children=o)

Python: Complete Program

import sys
from parsearg import ParseArg
from spark import (
    spark_start_cluster,
    spark_stop_cluster,
)
from hdfs import (
    hdfs_start,
    hdfs_stop,
)

view = {
    'spark|start': {
        'callback':  spark_start_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
    'spark|stop': {
        'callback':  spark_stop_cluster,
        '--instance': {
            'help':     'Spark cluster instance',
            'default':  'instance-1',
            'choices':   ['instance-1', 'instance-2'],
        },
    },
    'hdfs|start': {
        'callback':  hdfs_start,
        'help':     'Start Hadoop',
    },
    'hdfs|stop': {
        'callback':  hdfs_stop,
        'help':     'Stop Hadoop',
    },
}

def main(args: list)-> None:
    parser = ParseArg(view)
    ns     = parser.parse_args(args)
    result = ns.callback(ns)

if __name__ == "__main__":
    args = sys.argv[1:] if len(sys.argv) > 1 else []
    main(' '.join(args))

R: Complete Program

 #!/usr/bin/Rscript --no-init-file

suppressPackageStartupMessages(library(parsearg))

`spark_start_cluster`<- function(args) 
    cat(sprintf("starting Spark cluster instance=%s", args.instance)),
`spark_stop_cluster`<- function(args) 
    cat(sprintf("stopping Spark cluster instance=%s", args.instance)),
`hdfs_start`<- function(args) 
    cat(sprintf("starting HDFS cluster instance=%s", args.instance)),
`hdfs_stop`<- function(args) 
    cat(sprintf("stopping HDFS cluster instance=%s", args.instance)),

view = list(
    `spark|start|cluster` = list(
        `callback`    = spark_start_cluster,
        `--instance`  = list(
            `help`    =  'Spark cluster instance',
            `default` =  'instance-1',
            `choices` =   c('instance-1', 'instance-2')
        )
    ),
    `spark|stop|cluster` = list(
        `callback`    = spark_stop_cluster,
        `--instance`  = list(
            `help`    =  'Spark cluster instance',
            `default` =  'instance-1',
            `choices` =   c('instance-1', 'instance-2')
        )
    ),
    `hdfs|start` = list(
        `callback`    = hdfs_start,
        `--instance`  = list(
            `help`    =  'Start Hadoop'
        )
    ),
    `hdfs|stop` = list(
        `callback`    = hdfs_stop,
        `--instance`  = list(
            `help`    =  'Stop Hadoop'
        )
    )
)

`main`<- function(args=commandArgs(trailingOnly=TRUE)) {
    parser<- ParseArg(view)
    ns<-     parser$parse_args(args)
    result<- ns$callback(ns)
}
main()

Parting thought

"We are tech cockroaches...

When the next technology apocalypse comes, we'll be the only ones to survive!"

Michael Kane (Parlor West Loop, Chicago, 2022-06-03)

Treezer Goode

The Curious Tale of a Wood that Needed the Trees

Thomas P. Harte

Saturday, 2022-06-04

Disclaimer

Disclaimer

The Noble

Command Line Interface

Address to the Nation

on

The State of the CLI

Depends on the OS/language

But…

A bit of a mess

Depends on the OS/language

GNU

GNU: Git

Observation: DSL

GNU

GNU

GNU: Git subcommands

Observation: Recursion

GNU: Git subcommand

Observation: Treezer Goode

Þ Auld PyÞone

Python…

Python: argparse

Python: subcommands with argparse

Python: subcommands with argparse (muchos fun!)

Python: muchos fun :(

Python: subcommands with argparse

Observation: Imperative

Python: The CLI Renaissance

Python: plac

Python: click

C/C++

C: argparse

C++: ArgumentParser

Java: picocli

Python-le-Nouveau

Subcommands & options

Python: pure data

Python: pure data

Python: pure data

Python: pure data

Observation: MVC

Python: View (pure data)

Python: View (pure data)

Python: View (pure data)

Python: complete program

Subcommands & options

Subcommands: reordered

Subcommands: reordered

Python: View—original order

Python: View—reordered subcommands

R

Subcommands & options

R: View (pure data)

R: View (pure data)

R: View (pure data)

R: View (pure data)

R: View (pure data)

R: View (pure data)

R: complete program

R: complete program (alt)

Observations

University of CLI

UCLI @ DSL

argparse: Python

argparse: R

parsearg: Python, R

Canonical example

Command-line arguments

The essence of an argument parser

The essence of an argument parser

The essence of an argument parser

The essence of an argument parser

The essence of an argument parser

Command-line arguments: Unnamed

Command-line arguments: Unnamed

Python: `argparse`

Python: subcommands with `argparse`

Python: subcommands with `argparse` (muchos fun!)

Python: subcommands with `argparse`

Python: `plac`

Python: `click`

C: `argparse`

C++: `ArgumentParser`

Java: `picocli`

`argparse`: Python

`argparse`: R

`parsearg`: Python, R

Arguments to `add_arguments`

`parsearg`

Helper functions: `Key`

Helper functions: `Key`

Helper functions: `Key`

Helper functions: `Node`

Helper functions: `Node`

Helper functions: `Key` to `Node`

Helper functions: What does a `Node` do?

Helper functions: Shifting a `Key`

`parsearg`: The Guts