Merge pull request #1598 from xonsh/m

Function Macros

docs/index.rst

@@ -99,6 +99,7 @@ Contents
 
     tutorial
     tutorial_hist
+    tutorial_macros
     tutorial_xontrib
     tutorial_events
     tutorial_completers

docs/tutorial_macros.rst

@@ -0,0 +1,350 @@
+.. _tutorial_macros:
+
+************************************
+Tutorial: Macros
+************************************
+Bust out your DSLRs, people. It is time to closely examine macros!
+
+What are macro instructions?
+============================
+In generic terms, a programming macro is a special kind of syntax that
+replaces a smaller amount of code with a larger expression, syntax tree,
+code object, etc after the macro has been evaluated.
+In practice, macros pause the normal parsing and evaluation of the code
+that they contain. This is so that they can perform their expansion with
+a complete inputs. Roughly, the algorithm executing a macro follows is:
+
+1. Macro start, pause or skip normal parsing
+2. Gather macro inputs as strings
+3. Evaluate macro with inputs
+4. Resume normal parsing and execution.
+
+Is this meta-programming? You betcha!
+
+When and where are macros used?
+===============================
+Macros are a practicality-beats-purity feature of many programing
+languages. Because they allow you break out of the normal parsing
+cycle, depending on the language, you can do some truly wild things with
+them. However, macros are really there to reduce the amount of boiler plate
+code that users and developers have to write.
+
+In C and C++ (and Fortran), the C Preprocessor ``cpp`` is a macro evaluation
+engine. For example, every time you see an ``#include`` or ``#ifdef``, this is
+the ``cpp`` macro system in action.
+In these languages, the macros are technically outside of the definition
+of the language at hand. Furthermore, because ``cpp`` must function with only
+a single pass through the code, the sorts of macros that can be written with
+``cpp`` are relatively simple.
+
+Rust, on the other hand, has a first-class notion of macros that look and
+feel a lot like normal functions. Macros in Rust are capable of pulling off
+type information from their arguments and preventing their return values
+from being consumed.
+
+Other languages like Lisp, Forth, and Julia also provide their macro systems.
+Even restructured text (rST) directives could be considered macros.
+Haskell and other more purely functional languages do not need macros (since
+evaluation is lazy anyway), and so do not have them.
+
+If these seem unfamiliar to the Python world, note that Jupyter and IPython
+magics ``%`` and ``%%`` are macros!
+
+Function Macros
+===============
+Xonsh supports Rust-like macros that are based on normal Python callables.
+Macros do not require a special definition in xonsh. However, like in Rust,
+they must be called with an exclamation point ``!`` between the callable
+and the opening parentheses ``(``. Macro arguments are split on the top-level
+commas ``,``, like normal Python functions.  For example, say we have the
+functions ``f`` and ``g``. We could perform a macro call on these functions
+with the following:
+
+.. code-block:: xonsh
+
+    # No macro args
+    f!()
+
+    # Single arg
+    f!(x)
+    g!([y, 43, 44])
+
+    # Two args
+    f!(x, x + 42)
+    g!([y, 43, 44], f!(z))
+
+Not so bad, right?  So what actually happens to the arguments when used
+in a macro call?  Well, that depends on the definition of the function. In
+particular, each argument in the macro call is matched up with the corresponding
+parameter annotation in the callable's signature.  For example, say we have
+an ``identity()`` function that annotates its sole argument as a string:
+
+.. code-block:: xonsh
+
+    def identity(x : str):
+        return x
+
+If we call this normally, we'll just get whatever object we put in back out,
+even if that object is not a string:
+
+.. code-block:: xonshcon
+
+    >>> identity('me')
+    'me'
+
+    >>> identity(42)
+    42
+
+    >>> identity(identity)
+    <function __main__.identity>
+
+However, if we perform macro calls instead we are now guaranteed to get
+the string of the source code that is in the macro call:
+
+.. code-block:: xonshcon
+
+    >>> identity!('me')
+    "'me'"
+
+    >>> identity!(42)
+    '42'
+
+    >>> identity!(identity)
+    'identity'
+
+Also note that each macro argument is stripped prior to passing it to the
+macro itself. This is done for consistency.
+
+.. code-block:: xonshcon
+
+    >>> identity!(42)
+    '42'
+
+    >>> identity!(  42 )
+    '42'
+
+Importantly, because we are capturing and not evaluating the source code,
+a macro call can contain input that is beyond the usual syntax. In fact, that
+is sort of the whole point. Here are some cases to start your gears turning:
+
+.. code-block:: xonshcon
+
+    >>> identity!(import os)
+    'import os'
+
+    >>> identity!(if True:
+    >>>     pass)
+    'if True:\n    pass'
+
+    >>> identity!(std::vector<std::string> x = {"yoo", "hoo"})
+    'std::vector<std::string> x = {"yoo", "hoo"}'
+
+You do you, ``identity()``.
+
+Calling Function Macros
+=======================
+There are a couple of points to consider when calling macros. The first is
+that passing in arguments by name will not behave as expected. This is because
+the ``<name>=`` is captured by the macro itself. Using the ``identity()``
+function from above:
+
+.. code-block:: xonshcon
+
+    >>> identity!(x=42)
+    'x=42'
+
+Performing a macro call uses only argument order to pass in values.
+
+Additionally, macro calls split arguments only on the top-level commas.
+The top-level commas are not included in any argument.
+This behaves analogously to normal Python function calls. For instance,
+say we have the following ``g()`` function that accepts two arguments:
+
+.. code-block:: xonsh
+
+    def g(x : str, y : str):
+        print('x = ' + repr(x))
+        print('y = ' + repr(y))
+
+Then you can see the splitting and stripping behavior on each macro
+argument:
+
+.. code-block:: xonshcon
+
+    >>> g!(42, 65)
+    x = '42'
+    y = '65'
+
+    >>> g!(42, 65,)
+    x = '42'
+    y = '65'
+
+    >>> g!( 42, 65, )
+    x = '42'
+    y = '65'
+
+    >>> g!(['x', 'y'], {1: 1, 2: 3})
+    x = "['x', 'y']"
+    y = '{1: 1, 2: 3}'
+
+Sometimes you may only want to pass in the first few arguments as macro
+arguments and you want the rest to be treated as normal Python arguments.
+By convention, xonsh's macro caller will look for a lone ``*`` argument
+in order to split the macro arguments and the regular arguments. So for
+example:
+
+.. code-block:: xonshcon
+
+    >>> g!(42, *, 65)
+    x = '42'
+    y = 65
+
+    >>> g!(42, *, y=65)
+    x = '42'
+    y = 65
+
+In the above, note that ``x`` is still captured as a macro argument. However,
+everything after the ``*``, namely ``y``, is evaluated is if it were passed
+in to a normal function call.  This can be useful for large interfaces where
+only a handful of args are expected as macro arguments.
+
+Hopefully, now you see the big picture.
+
+Writing Function Macros
+=======================
+Though any function (or callable) can be used as a macro, this functionality
+is probably most useful if the function was *designed* as a macro. There
+are two main aspects of macro design to consider: argument annotations and
+call site execution context.
+
+
+Macro Function Argument Annotations
+-----------------------------------
+There are six kinds of annotations that macros are able to interpret:
+
+.. list-table:: Kinds of Annotation
+   :header-rows: 1
+
+   * - Category
+     - Object
+     - Flags
+     - Modes
+     - Returns
+   * - String
+     - ``str``
+     - ``'s'``, ``'str'``, or ``'string'``
+     -
+     - Source code of argument as string.
+   * - AST
+     - ``ast.AST``
+     - ``'a'`` or ``'ast'``
+     - ``'eval'`` (default), ``'exec'``, or ``'single'``
+     - Abstract syntax tree of argument.
+   * - Code
+     - ``types.CodeType`` or ``compile``
+     - ``'c'``, ``'code'``, or ``'compile'``
+     - ``'eval'`` (default), ``'exec'``, or ``'single'``
+     - Compiled code object of argument.
+   * - Eval
+     - ``eval`` or ``None``
+     - ``'v'`` or ``'eval'``
+     -
+     - Evaluation of the argument, *default*.
+   * - Exec
+     - ``exec``
+     - ``'x'`` or ``'exec'``
+     - ``'exec'`` (default) or ``'single'``
+     - Execs the argument and returns None.
+   * - Type
+     - ``type``
+     - ``'t'`` or ``'type'``
+     -
+     - The type of the argument after it has been evaluated.
+
+These annotations allow you to hook into whichever stage of the compilation
+that you desire. It is important to note that the string form of the arguments
+is split and stripped (as described above) prior to conversion to the
+annotation type.
+
+Each argument may be annotated with its own individual type. Annotations
+may be provided as either objects or as the string flags seen in the above
+table. String flags are case-insensitive.
+If an argument does not have an annotation, ``eval`` is selected.
+This makes the macro call behave like a normal function call for
+arguments whose annotations are unspecified.  For example,
+
+.. code-block:: xonsh
+
+    def func(a, b : 'AST', c : compile):
+        pass
+
+In a macro call of ``func!()``,
+
+* ``a`` will be evaluated with ``eval`` since no annotation was provided,
+* ``b`` will be parsed into a syntax tree node, and
+* ``c`` will be compiled into code object since the builtin ``compile()``
+  function was used as the annotation.
+
+Additionally, certain kinds of annotations have different modes that
+affect the parsing, compilation, and execution of its argument.  While a
+sensible default is provided, you may also supply your own. This is
+done by annotating with a (kind, mode) tuple.  The first element can
+be any valid object or flag. The second element must be a corresponding
+mode as a string.  For instance,
+
+.. code-block:: xonsh
+
+    def gunc(d : (exec, 'single'), e : ('c', 'exec')):
+        pass
+
+Thus in a macro call of ``gunc!()``,
+
+* ``d`` will be exec'd in single-mode (rather than exec-mode), and
+* ``e`` will be compiled in exec-mode (rather than eval-mode).
+
+For more information on the differences between the exec, eval, and single
+modes please see the Python documentation.
+
+
+Macro Function Execution Context
+--------------------------------
+Equally important as having the macro arguments is knowing the execution
+context of the macro call itself. Rather than mucking around with frames,
+macros provide both the globals and locals of the call site.  These are
+accessible as the ``macro_globals`` and ``macro_locals`` attributes of
+the macro function itself while the macro is being executed.
+
+For example, consider a macro which replaces all literal ``1`` digits
+with the literal ``2``, evaluates the modification, and returns the results.
+To eval, the macro will need to pull off its globals and locals:
+
+.. code-block:: xonsh
+
+    def one_to_two(x : str):
+        s = x.replace('1', '2')
+        glbs = one_to_two.macro_globals
+        locs = one_to_two.macro_locals
+        return eval(s, glbs, locs)
+
+Running this with a few of different inputs, we see:
+
+.. code-block:: xonshcon
+
+    >>> one_to_two!(1 + 1)
+    4
+
+    >>> one_to_two!(11)
+    22
+
+    >>> x = 1
+    >>> one_to_two!(x + 1)
+    3
+
+Of course, many other more sophisticated options are available depending on the
+use case.
+
+
+Take Away
+=========
+Hopefully, at this point, you see that a few well placed macros can be extremely
+convenient and valuable to any project.

news/m.rst

@@ -0,0 +1,14 @@
+**Added:**
+
+* Macro function calls are now available. These use a Rust-like
+  ``f!(arg)`` syntax.
+
+**Changed:** None
+
+**Deprecated:** None
+
+**Removed:** None
+
+**Fixed:** None
+
+**Security:** None

setup.cfg

@@ -5,7 +5,7 @@ flake8-ignore =
     *.py E402
     tests/tools.py E128
     xonsh/ast.py F401
-    xonsh/built_ins.py F821
+    xonsh/built_ins.py F821 E721
     xonsh/commands_cache.py F841
     xonsh/history.py F821
     xonsh/pyghooks.py F821

tests/test_builtins.py

@@ -3,13 +3,16 @@
 from __future__ import unicode_literals, print_function
 import os
 import re
+import builtins
+import types
+from ast import AST
 
 import pytest
 
 from xonsh import built_ins
 from xonsh.built_ins import reglob, pathsearch, helper, superhelper, \
     ensure_list_of_strs, list_of_strs_or_callables, regexsearch, \
-    globsearch
+    globsearch, convert_macro_arg, macro_context, call_macro
 from xonsh.environ import Env
 
 from tools import skip_if_on_windows
@@ -18,6 +21,10 @@ from tools import skip_if_on_windows
 HOME_PATH = os.path.expanduser('~')
 
 
+@pytest.fixture(autouse=True)
+def xonsh_execer_autouse(xonsh_execer):
+    return xonsh_execer
+
 @pytest.mark.parametrize('testfile', reglob('test_.*'))
 def test_reglob_tests(testfile):
     assert (testfile.startswith('test_'))
@@ -113,3 +120,146 @@ f = lambda x: 20
 def test_list_of_strs_or_callables(exp, inp):
     obs = list_of_strs_or_callables(inp)
     assert exp == obs
+
+
+@pytest.mark.parametrize('kind', [str, 's', 'S', 'str', 'string'])
+def test_convert_macro_arg_str(kind):
+    raw_arg = 'value'
+    arg = convert_macro_arg(raw_arg, kind, None, None)
+    assert arg is raw_arg
+
+
+@pytest.mark.parametrize('kind', [AST, 'a', 'Ast'])
+def test_convert_macro_arg_ast(kind):
+    raw_arg = '42'
+    arg = convert_macro_arg(raw_arg, kind, {}, None)
+    assert isinstance(arg, AST)
+
+
+@pytest.mark.parametrize('kind', [types.CodeType, compile, 'c', 'code',
+                                  'compile'])
+def test_convert_macro_arg_code(kind):
+    raw_arg = '42'
+    arg = convert_macro_arg(raw_arg, kind, {}, None)
+    assert isinstance(arg, types.CodeType)
+
+
+@pytest.mark.parametrize('kind', [eval, None, 'v', 'eval'])
+def test_convert_macro_arg_eval(kind):
+    # literals
+    raw_arg = '42'
+    arg = convert_macro_arg(raw_arg, kind, {}, None)
+    assert arg == 42
+    # exprs
+    raw_arg = 'x + 41'
+    arg = convert_macro_arg(raw_arg, kind, {}, {'x': 1})
+    assert arg == 42
+
+
+@pytest.mark.parametrize('kind', [exec, 'x', 'exec'])
+def test_convert_macro_arg_exec(kind):
+    # at global scope
+    raw_arg = 'def f(x, y):\n    return x + y'
+    glbs = {}
+    arg = convert_macro_arg(raw_arg, kind, glbs, None)
+    assert arg is None
+    assert 'f' in glbs
+    assert glbs['f'](1, 41) == 42
+    # at local scope
+    raw_arg = 'def g(z):\n    return x + z\ny += 42'
+    glbs = {'x': 40}
+    locs = {'y': 1}
+    arg = convert_macro_arg(raw_arg, kind, glbs, locs)
+    assert arg is None
+    assert 'g' in locs
+    assert locs['g'](1) == 41
+    assert 'y' in locs
+    assert locs['y'] == 43
+
+
+@pytest.mark.parametrize('kind', [type, 't', 'type'])
+def test_convert_macro_arg_eval(kind):
+    # literals
+    raw_arg = '42'
+    arg = convert_macro_arg(raw_arg, kind, {}, None)
+    assert arg is int
+    # exprs
+    raw_arg = 'x + 41'
+    arg = convert_macro_arg(raw_arg, kind, {}, {'x': 1})
+    assert arg is int
+
+
+def test_macro_context():
+    def f():
+        pass
+    with macro_context(f, True, True):
+        assert f.macro_globals
+        assert f.macro_locals
+    assert not hasattr(f, 'macro_globals')
+    assert not hasattr(f, 'macro_locals')
+
+
+@pytest.mark.parametrize('arg', ['x', '42', 'x + y'])
+def test_call_macro_str(arg):
+    def f(x : str):
+        return x
+    rtn = call_macro(f, [arg], None, None)
+    assert rtn is arg
+
+
+@pytest.mark.parametrize('arg', ['x', '42', 'x + y'])
+def test_call_macro_ast(arg):
+    def f(x : AST):
+        return x
+    rtn = call_macro(f, [arg], {}, None)
+    assert isinstance(rtn, AST)
+
+
+@pytest.mark.parametrize('arg', ['x', '42', 'x + y'])
+def test_call_macro_code(arg):
+    def f(x : compile):
+        return x
+    rtn = call_macro(f, [arg], {}, None)
+    assert isinstance(rtn, types.CodeType)
+
+
+@pytest.mark.parametrize('arg', ['x', '42', 'x + y'])
+def test_call_macro_eval(arg):
+    def f(x : eval):
+        return x
+    rtn = call_macro(f, [arg], {'x': 42, 'y': 0}, None)
+    assert rtn == 42
+
+
+@pytest.mark.parametrize('arg', ['if y:\n    pass',
+                                 'if 42:\n    pass',
+                                 'if x + y:\n    pass'])
+def test_call_macro_exec(arg):
+    def f(x : exec):
+        return x
+    rtn = call_macro(f, [arg], {'x': 42, 'y': 0}, None)
+    assert rtn is None
+
+
+@pytest.mark.parametrize('arg', ['x', '42', 'x + y'])
+def test_call_macro_raw_arg(arg):
+    def f(x : str):
+        return x
+    rtn = call_macro(f, ['*', arg], {'x': 42, 'y': 0}, None)
+    assert rtn == 42
+
+
+@pytest.mark.parametrize('arg', ['x', '42', 'x + y'])
+def test_call_macro_raw_kwarg(arg):
+    def f(x : str):
+        return x
+    rtn = call_macro(f, ['*', 'x=' + arg], {'x': 42, 'y': 0}, None)
+    assert rtn == 42
+
+
+@pytest.mark.parametrize('arg', ['x', '42', 'x + y'])
+def test_call_macro_raw_kwargs(arg):
+    def f(x : str):
+        return x
+    rtn = call_macro(f, ['*', '**{"x" :' + arg + '}'], {'x': 42, 'y': 0}, None)
+    assert rtn == 42

tests/test_parser.py

@@ -4,10 +4,11 @@ import os
 import sys
 import ast
 import builtins
+import itertools
 
 import pytest
 
-from xonsh.ast import pdump
+from xonsh.ast import pdump, AST
 from xonsh.parser import Parser
 
 from tools import VER_FULL, skip_if_py34, nodes_equal
@@ -42,16 +43,17 @@ def check_stmts(inp, run=True, mode='exec'):
         inp += '\n'
     check_ast(inp, run=run, mode=mode)
 
-def check_xonsh_ast(xenv, inp, run=True, mode='eval'):
+def check_xonsh_ast(xenv, inp, run=True, mode='eval', debug_level=0,
+                    return_obs=False):
     __tracebackhide__ = True
     builtins.__xonsh_env__ = xenv
-    obs = PARSER.parse(inp)
+    obs = PARSER.parse(inp, debug_level=debug_level)
     if obs is None:
         return  # comment only
     bytecode = compile(obs, '<test-xonsh-ast>', mode)
     if run:
         exec(bytecode)
-    return True
+    return obs if return_obs else True
 
 def check_xonsh(xenv, inp, run=True, mode='exec'):
     __tracebackhide__ = True
@@ -1796,6 +1798,74 @@ def test_redirect_error_to_output(r, o):
     assert check_xonsh_ast({}, '$[< input.txt echo "test" {} {}> test.txt]'.format(r, o), False)
     assert check_xonsh_ast({}, '$[echo "test" {} {}> test.txt < input.txt]'.format(r, o), False)
 
+def test_macro_call_empty():
+    assert check_xonsh_ast({}, 'f!()', False)
+
+
+MACRO_ARGS = [
+    'x', 'True', 'None', 'import os', 'x=10', '"oh no, mom"', '...', ' ... ',
+    'if True:\n  pass', '{x: y}', '{x: y, 42: 5}', '{1, 2, 3,}', '(x,y)',
+    '(x, y)', '((x, y), z)', 'g()', 'range(10)', 'range(1, 10, 2)', '()', '{}',
+    '[]', '[1, 2]', '@(x)', '!(ls -l)', '![ls -l]', '$(ls -l)', '${x + y}',
+    '$[ls -l]', '@$(which xonsh)',
+]
+
+@pytest.mark.parametrize('s', MACRO_ARGS)
+def test_macro_call_one_arg(s):
+    f = 'f!({})'.format(s)
+    tree = check_xonsh_ast({}, f, False, return_obs=True)
+    assert isinstance(tree, AST)
+    args = tree.body.args[1].elts
+    assert len(args) == 1
+    assert args[0].s == s.strip()
+
+
+@pytest.mark.parametrize('s,t', itertools.product(MACRO_ARGS[::2],
+                                                  MACRO_ARGS[1::2]))
+def test_macro_call_two_args(s, t):
+    f = 'f!({}, {})'.format(s, t)
+    tree = check_xonsh_ast({}, f, False, return_obs=True)
+    assert isinstance(tree, AST)
+    args = tree.body.args[1].elts
+    assert len(args) == 2
+    assert args[0].s == s.strip()
+    assert args[1].s == t.strip()
+
+
+@pytest.mark.parametrize('s,t,u', itertools.product(MACRO_ARGS[::3],
+                                                    MACRO_ARGS[1::3],
+                                                    MACRO_ARGS[2::3]))
+def test_macro_call_three_args(s, t, u):
+    f = 'f!({}, {}, {})'.format(s, t, u)
+    tree = check_xonsh_ast({}, f, False, return_obs=True)
+    assert isinstance(tree, AST)
+    args = tree.body.args[1].elts
+    assert len(args) == 3
+    assert args[0].s == s.strip()
+    assert args[1].s == t.strip()
+    assert args[2].s == u.strip()
+
+
+@pytest.mark.parametrize('s', MACRO_ARGS)
+def test_macro_call_one_trailing(s):
+    f = 'f!({0},)'.format(s)
+    tree = check_xonsh_ast({}, f, False, return_obs=True)
+    assert isinstance(tree, AST)
+    args = tree.body.args[1].elts
+    assert len(args) == 1
+    assert args[0].s == s.strip()
+
+
+@pytest.mark.parametrize('s', MACRO_ARGS)
+def test_macro_call_one_trailing_space(s):
+    f = 'f!( {0}, )'.format(s)
+    tree = check_xonsh_ast({}, f, False, return_obs=True)
+    assert isinstance(tree, AST)
+    args = tree.body.args[1].elts
+    assert len(args) == 1
+    assert args[0].s == s.strip()
+
+
 # test invalid expressions
 
 def test_syntax_error_del_literal():

xonsh/built_ins.py

@@ -8,16 +8,19 @@ import os
 import re
 import sys
 import time
+import types
 import shlex
 import signal
 import atexit
 import inspect
 import tempfile
 import builtins
+import itertools
 import subprocess
 import contextlib
 import collections.abc as cabc
 
+from xonsh.ast import AST
 from xonsh.lazyasd import LazyObject, lazyobject
 from xonsh.history import History
 from xonsh.inspectors import Inspector
@@ -680,6 +683,197 @@ def list_of_strs_or_callables(x):
     return rtn
 
 
+@lazyobject
+def MACRO_FLAG_KINDS():
+    return {
+        's': str,
+        'str': str,
+        'string': str,
+        'a': AST,
+        'ast': AST,
+        'c': types.CodeType,
+        'code': types.CodeType,
+        'compile': types.CodeType,
+        'v': eval,
+        'eval': eval,
+        'x': exec,
+        'exec': exec,
+        't': type,
+        'type': type,
+        }
+
+
+def _convert_kind_flag(x):
+    """Puts a kind flag (string) a canonical form."""
+    x = x.lower()
+    kind = MACRO_FLAG_KINDS.get(x, None)
+    if kind is None:
+        raise TypeError('{0!r} not a recognized macro type.'.format(x))
+    return kind
+
+
+def convert_macro_arg(raw_arg, kind, glbs, locs, *, name='<arg>',
+                      macroname='<macro>'):
+    """Converts a string macro argument based on the requested kind.
+
+    Parameters
+    ----------
+    raw_arg : str
+        The str reprensetaion of the macro argument.
+    kind : object
+        A flag or type representing how to convert the argument.
+    glbs : Mapping
+        The globals from the call site.
+    locs : Mapping or None
+        The locals from the call site.
+    name : str, optional
+        The macro argument name.
+    macroname : str, optional
+        The name of the macro itself.
+
+    Returns
+    -------
+    The converted argument.
+    """
+    # munge kind and mode to start
+    mode = None
+    if isinstance(kind, cabc.Sequence) and not isinstance(kind, str):
+        # have (kind, mode) tuple
+        kind, mode = kind
+    if isinstance(kind, str):
+        kind = _convert_kind_flag(kind)
+    if kind is str:
+        return raw_arg  # short circut since there is nothing else to do
+    # select from kind and convert
+    execer = builtins.__xonsh_execer__
+    filename = macroname + '(' + name + ')'
+    if kind is AST:
+        ctx = set(dir(builtins)) | set(glbs.keys())
+        if locs is not None:
+            ctx |= set(locs.keys())
+        mode = mode or 'eval'
+        arg = execer.parse(raw_arg, ctx, mode=mode, filename=filename)
+    elif kind is types.CodeType or kind is compile:
+        mode = mode or 'eval'
+        arg = execer.compile(raw_arg, mode=mode, glbs=glbs, locs=locs,
+                             filename=filename)
+    elif kind is eval or kind is None:
+        arg = execer.eval(raw_arg, glbs=glbs, locs=locs, filename=filename)
+    elif kind is exec:
+        mode = mode or 'exec'
+        if not raw_arg.endswith('\n'):
+            raw_arg += '\n'
+        arg = execer.exec(raw_arg, mode=mode, glbs=glbs, locs=locs,
+                          filename=filename)
+    elif kind is type:
+        arg = type(execer.eval(raw_arg, glbs=glbs, locs=locs,
+                               filename=filename))
+    else:
+        msg = ('kind={0!r} and mode={1!r} was not recongnized for macro '
+               'argument {2!r}')
+        raise TypeError(msg.format(kind, mode, name))
+    return arg
+
+
+@contextlib.contextmanager
+def macro_context(f, glbs, locs):
+    """Attaches macro globals and locals temporarily to function as a
+    context manager.
+
+    Parameters
+    ----------
+    f : callable object
+        The function that is called as f(*args).
+    glbs : Mapping
+        The globals from the call site.
+    locs : Mapping or None
+        The locals from the call site.
+    """
+    prev_glbs = getattr(f, 'macro_globals', None)
+    prev_locs = getattr(f, 'macro_locals', None)
+    f.macro_globals = glbs
+    f.macro_locals = locs
+    yield
+    if prev_glbs is None:
+        del f.macro_globals
+    else:
+        f.macro_globals = prev_glbs
+    if prev_locs is None:
+        del f.macro_locals
+    else:
+        f.macro_locals = prev_locs
+
+
+def call_macro(f, raw_args, glbs, locs):
+    """Calls a function as a macro, returning its result.
+
+    Parameters
+    ----------
+    f : callable object
+        The function that is called as f(*args).
+    raw_args : tuple of str
+        The str reprensetaion of arguments of that were passed into the
+        macro. These strings will be parsed, compiled, evaled, or left as
+        a string dependending on the annotations of f.
+    glbs : Mapping
+        The globals from the call site.
+    locs : Mapping or None
+        The locals from the call site.
+    """
+    sig = inspect.signature(f)
+    empty = inspect.Parameter.empty
+    macroname = f.__name__
+    i = 0
+    args = []
+    for (key, param), raw_arg in zip(sig.parameters.items(), raw_args):
+        i += 1
+        if raw_arg == '*':
+            break
+        kind = param.annotation
+        if kind is empty or kind is None:
+            kind = eval
+        arg = convert_macro_arg(raw_arg, kind, glbs, locs, name=key,
+                                macroname=macroname)
+        args.append(arg)
+    reg_args, kwargs = _eval_regular_args(raw_args[i:], glbs, locs)
+    args += reg_args
+    with macro_context(f, glbs, locs):
+        rtn = f(*args, **kwargs)
+    return rtn
+
+
+@lazyobject
+def KWARG_RE():
+    return re.compile('([A-Za-z_]\w*=|\*\*)')
+
+
+def _starts_as_arg(s):
+    """Tests if a string starts as a non-kwarg string would."""
+    return KWARG_RE.match(s) is None
+
+
+def _eval_regular_args(raw_args, glbs, locs):
+    if not raw_args:
+        return [], {}
+    arglist = list(itertools.takewhile(_starts_as_arg, raw_args))
+    kwarglist = raw_args[len(arglist):]
+    execer = builtins.__xonsh_execer__
+    if not arglist:
+        args = arglist
+        kwargstr = 'dict({})'.format(', '.join(kwarglist))
+        kwargs = execer.eval(kwargstr, glbs=glbs, locs=locs)
+    elif not kwarglist:
+        argstr = '({},)'.format(', '.join(arglist))
+        args = execer.eval(argstr, glbs=glbs, locs=locs)
+        kwargs = {}
+    else:
+        argstr = '({},)'.format(', '.join(arglist))
+        kwargstr = 'dict({})'.format(', '.join(kwarglist))
+        both = '({}, {})'.format(argstr, kwargstr)
+        args, kwargs = execer.eval(both, glbs=glbs, locs=locs)
+    return args, kwargs
+
+
 def load_builtins(execer=None, config=None, login=False, ctx=None):
     """Loads the xonsh builtins into the Python builtins. Sets the
     BUILTINS_LOADED variable to True.
@@ -715,7 +909,7 @@ def load_builtins(execer=None, config=None, login=False, ctx=None):
     builtins.__xonsh_ensure_list_of_strs__ = ensure_list_of_strs
     builtins.__xonsh_list_of_strs_or_callables__ = list_of_strs_or_callables
     builtins.__xonsh_completers__ = xonsh.completers.init.default_completers()
-    builtins.events = events
+    builtins.__xonsh_call_macro__ = call_macro
     # public built-ins
     builtins.XonshError = XonshError
     builtins.XonshBlockError = XonshBlockError
@@ -723,6 +917,7 @@ def load_builtins(execer=None, config=None, login=False, ctx=None):
     builtins.evalx = None if execer is None else execer.eval
     builtins.execx = None if execer is None else execer.exec
     builtins.compilex = None if execer is None else execer.compile
+    builtins.events = events
 
     # sneak the path search functions into the aliases
     # Need this inline/lazy import here since we use locate_binary that relies on __xonsh_env__ in default aliases
@@ -781,6 +976,7 @@ def unload_builtins():
              '__xonsh_execer__',
              '__xonsh_commands_cache__',
              '__xonsh_completers__',
+             '__xonsh_call_macro__',
              'XonshError',
              'XonshBlockError',
              'XonshCalledProcessError',

xonsh/execer.py

@@ -45,13 +45,15 @@ class Execer(object):
         if self.unload:
             unload_builtins()
 
-    def parse(self, input, ctx, mode='exec', transform=True):
+    def parse(self, input, ctx, mode='exec', filename=None, transform=True):
         """Parses xonsh code in a context-aware fashion. For context-free
         parsing, please use the Parser class directly or pass in
         transform=False.
         """
+        if filename is None:
+            filename = self.filename
         if not transform:
-            return self.parser.parse(input, filename=self.filename, mode=mode,
+            return self.parser.parse(input, filename=filename, mode=mode,
                                      debug_level=(self.debug_level > 1))
 
         # Parsing actually happens in a couple of phases. The first is a
@@ -68,7 +70,7 @@ class Execer(object):
         # tokens for all of the Python rules. The lazy way implemented here
         # is to parse a line a second time with a $() wrapper if it fails
         # the first time. This is a context-free phase.
-        tree, input = self._parse_ctx_free(input, mode=mode)
+        tree, input = self._parse_ctx_free(input, mode=mode, filename=filename)
         if tree is None:
             return None
 
@@ -97,7 +99,8 @@ class Execer(object):
             glbs = frame.f_globals if glbs is None else glbs
             locs = frame.f_locals if locs is None else locs
         ctx = set(dir(builtins)) | set(glbs.keys()) | set(locs.keys())
-        tree = self.parse(input, ctx, mode=mode, transform=transform)
+        tree = self.parse(input, ctx, mode=mode, filename=filename,
+                          transform=transform)
         if tree is None:
             return None  # handles comment only input
         if transform:
@@ -110,45 +113,53 @@ class Execer(object):
         return code
 
     def eval(self, input, glbs=None, locs=None, stacklevel=2,
-             transform=True):
+             filename=None, transform=True):
         """Evaluates (and returns) xonsh code."""
         if isinstance(input, types.CodeType):
             code = input
         else:
+            if filename is None:
+                filename = self.filename
             code = self.compile(input=input,
                                 glbs=glbs,
                                 locs=locs,
                                 mode='eval',
                                 stacklevel=stacklevel,
+                                filename=filename,
                                 transform=transform)
         if code is None:
             return None  # handles comment only input
         return eval(code, glbs, locs)
 
     def exec(self, input, mode='exec', glbs=None, locs=None, stacklevel=2,
-             transform=True):
+             filename=None, transform=True):
         """Execute xonsh code."""
         if isinstance(input, types.CodeType):
             code = input
         else:
+            if filename is None:
+                filename = self.filename
             code = self.compile(input=input,
                                 glbs=glbs,
                                 locs=locs,
                                 mode=mode,
                                 stacklevel=stacklevel,
+                                filename=filename,
                                 transform=transform)
         if code is None:
             return None  # handles comment only input
         return exec(code, glbs, locs)
 
-    def _parse_ctx_free(self, input, mode='exec'):
+    def _parse_ctx_free(self, input, mode='exec', filename=None):
         last_error_line = last_error_col = -1
         parsed = False
         original_error = None
+        if filename is None:
+            filename = self.filename
         while not parsed:
             try:
                 tree = self.parser.parse(input,
-                                         filename=self.filename,
+                                         filename=filename,
                                          mode=mode,
                                          debug_level=(self.debug_level > 1))
                 parsed = True

xonsh/parsers/base.py

@@ -220,6 +220,10 @@ class BaseParser(object):
         self.lexer = lexer = Lexer()
         self.tokens = lexer.tokens
 
+        self._lines = None
+        self.xonsh_code = None
+        self._attach_nocomma_tok_rules()
+
         opt_rules = [
             'newlines', 'arglist', 'func_call', 'rarrow_test', 'typedargslist',
             'equals_test', 'colon_test', 'tfpdef', 'comma_tfpdef_list',
@@ -234,7 +238,8 @@ class BaseParser(object):
             'op_factor_list', 'trailer_list', 'testlist_comp',
             'yield_expr_or_testlist_comp', 'dictorsetmaker',
             'comma_subscript_list', 'test', 'sliceop', 'comp_iter',
-            'yield_arg', 'test_comma_list']
+            'yield_arg', 'test_comma_list',
+            'macroarglist', 'any_raw_toks']
         for rule in opt_rules:
             self._opt_rule(rule)
 
@@ -248,7 +253,7 @@ class BaseParser(object):
             'pm_term', 'op_factor', 'trailer', 'comma_subscript',
             'comma_expr_or_star_expr', 'comma_test', 'comma_argument',
             'comma_item', 'attr_period_name', 'test_comma',
-            'equals_yield_expr_or_testlist']
+            'equals_yield_expr_or_testlist', 'comma_nocomma']
         for rule in list_rules:
             self._list_rule(rule)
 
@@ -260,7 +265,7 @@ class BaseParser(object):
                      'for', 'colon', 'import', 'except', 'nonlocal', 'global',
                      'yield', 'from', 'raise', 'with', 'dollar_lparen',
                      'dollar_lbrace', 'dollar_lbracket', 'try',
-                     'bang_lparen', 'bang_lbracket']
+                     'bang_lparen', 'bang_lbracket', 'comma', 'rparen']
         for rule in tok_rules:
             self._tok_rule(rule)
 
@@ -288,6 +293,8 @@ class BaseParser(object):
         """Resets for clean parsing."""
         self.lexer.reset()
         self._last_yielded_token = None
+        self._lines = None
+        self.xonsh_code = None
 
     def parse(self, s, filename='<code>', mode='exec', debug_level=0):
         """Returns an abstract syntax tree of xonsh code.
@@ -324,7 +331,7 @@ class BaseParser(object):
         return tree
 
     def _lexer_errfunc(self, msg, line, column):
-        self._parse_error(msg, self.currloc(line, column), self.xonsh_code)
+        self._parse_error(msg, self.currloc(line, column))
 
     def _yacc_lookahead_token(self):
         """Gets the next-to-last and last token seen by the lexer."""
@@ -406,15 +413,33 @@ class BaseParser(object):
             return self.token_col(t)
         return 0
 
-    def _parse_error(self, msg, loc, line=None):
-        if line is None:
+    @property
+    def lines(self):
+        if self._lines is None and self.xonsh_code is not None:
+            self._lines = self.xonsh_code.splitlines(keepends=True)
+        return self._lines
+
+    def source_slice(self, start, stop):
+        """Gets the original source code from two (line, col) tuples in
+        source-space (i.e. lineno start at 1).
+        """
+        bline, bcol = start
+        eline, ecol = stop
+        bline -= 1
+        lines = self.lines[bline:eline]
+        lines[-1] = lines[-1][:ecol]
+        lines[0] = lines[0][bcol:]
+        return ''.join(lines)
+
+    def _parse_error(self, msg, loc):
+        if self.xonsh_code is None or loc is None:
             err_line_pointer = ''
         else:
             col = loc.column + 1
-            lines = line.splitlines()
+            lines = self.lines
             i = loc.lineno - 1
             if 0 <= i < len(lines):
-                err_line = lines[i]
+                err_line = lines[i].rstrip()
                 err_line_pointer = '\n{}\n{: >{}}'.format(err_line, '^', col)
             else:
                 err_line_pointer = ''
@@ -429,7 +454,8 @@ class BaseParser(object):
                   ('left', 'EQ', 'NE'), ('left', 'GT', 'GE', 'LT', 'LE'),
                   ('left', 'RSHIFT', 'LSHIFT'), ('left', 'PLUS', 'MINUS'),
                   ('left', 'TIMES', 'DIVIDE', 'DOUBLEDIV', 'MOD'),
-                  ('left', 'POW'), )
+                  ('left', 'POW'),
+                  )
 
     #
     # Grammar as defined by BNF
@@ -480,7 +506,8 @@ class BaseParser(object):
     def p_eval_input(self, p):
         """eval_input : testlist newlines_opt
         """
-        p[0] = ast.Expression(body=p[1])
+        p1 = p[1]
+        p[0] = ast.Expression(body=p1, lineno=p1.lineno, col_offset=p1.col_offset)
 
     def p_func_call(self, p):
         """func_call : LPAREN arglist_opt RPAREN"""
@@ -1644,9 +1671,19 @@ class BaseParser(object):
                                    lineno=leader.lineno,
                                    col_offset=leader.col_offset)
             elif isinstance(trailer, Mapping):
+                # call normal functions
                 p0 = ast.Call(func=leader,
                               lineno=leader.lineno,
                               col_offset=leader.col_offset, **trailer)
+            elif isinstance(trailer, (ast.Tuple, tuple)):
+                # call macro functions
+                l, c = leader.lineno, leader.col_offset
+                gblcall = xonsh_call('globals', [], lineno=l, col=c)
+                loccall = xonsh_call('locals', [], lineno=l, col=c)
+                if isinstance(trailer, tuple):
+                    trailer, arglist = trailer
+                margs = [leader, trailer, gblcall, loccall]
+                p0 = xonsh_call('__xonsh_call_macro__', margs, lineno=l, col=c)
             elif isinstance(trailer, str):
                 if trailer == '?':
                     p0 = xonsh_help(leader, lineno=leader.lineno,
@@ -1827,6 +1864,34 @@ class BaseParser(object):
         """trailer : LPAREN arglist_opt RPAREN"""
         p[0] = [p[2] or dict(args=[], keywords=[], starargs=None, kwargs=None)]
 
+    def p_trailer_bang_lparen(self, p):
+        """trailer : bang_lparen_tok macroarglist_opt rparen_tok
+                   | bang_lparen_tok nocomma comma_tok rparen_tok
+                   | bang_lparen_tok nocomma comma_tok WS rparen_tok
+                   | bang_lparen_tok macroarglist comma_tok rparen_tok
+                   | bang_lparen_tok macroarglist comma_tok WS rparen_tok
+        """
+        p1, p2, p3 = p[1], p[2], p[3]
+        begins = [(p1.lineno, p1.lexpos + 2)]
+        ends = [(p3.lineno, p3.lexpos)]
+        if p2:
+            begins.extend([(x[0], x[1] + 1) for x in p2])
+            ends = p2 + ends
+        elts = []
+        for beg, end in zip(begins, ends):
+            s = self.source_slice(beg, end).strip()
+            if not s:
+                if len(begins) == 1:
+                    break
+                else:
+                    msg = 'empty macro arguments not allowed'
+                    self._parse_error(msg, self.currloc(*beg))
+            node = ast.Str(s=s, lineno=beg[0], col_offset=beg[1])
+            elts.append(node)
+        p0 = ast.Tuple(elts=elts, ctx=ast.Load(), lineno=p1.lineno,
+                       col_offset=p1.lexpos)
+        p[0] = [p0]
+
     def p_trailer_p3(self, p):
         """trailer : LBRACKET subscriptlist RBRACKET
                    | PERIOD NAME
@@ -1839,6 +1904,81 @@ class BaseParser(object):
         """
         p[0] = [p[1]]
 
+    def _attach_nocomma_tok_rules(self):
+        toks = set(self.tokens)
+        toks -= {'COMMA', 'LPAREN', 'RPAREN', 'LBRACE', 'RBRACE', 'LBRACKET',
+                 'RBRACKET', 'AT_LPAREN', 'BANG_LPAREN', 'BANG_LBRACKET',
+                 'DOLLAR_LPAREN', 'DOLLAR_LBRACE', 'DOLLAR_LBRACKET',
+                 'ATDOLLAR_LPAREN'}
+        ts = '\n            | '.join(sorted(toks))
+        doc = 'nocomma_tok : ' + ts + '\n'
+        self.p_nocomma_tok.__func__.__doc__ = doc
+
+    # The following grammar rules are no-ops because we don't need to glue the
+    # source code back together piece-by-piece. Instead, we simply look for
+    # top-level commas and record their positions. With these positions and
+    # the bounding parantheses !() positions we can use the source_slice()
+    # method. This does a much better job of capturing exactly the source code
+    # that was provided. The tokenizer & lexer can be a little lossy, especially
+    # with respect to whitespace.
+
+    def p_nocomma_tok(self, p):
+        # see attachement function above for docstring
+        pass
+
+    def p_any_raw_tok(self, p):
+        """any_raw_tok : nocomma
+                       | COMMA
+        """
+        pass
+
+    def p_any_raw_toks_one(self, p):
+        """any_raw_toks : any_raw_tok"""
+        pass
+
+    def p_any_raw_toks_many(self, p):
+        """any_raw_toks : any_raw_toks any_raw_tok"""
+        pass
+
+    def p_nocomma_part_tok(self, p):
+        """nocomma_part : nocomma_tok"""
+        pass
+
+    def p_nocomma_part_any(self, p):
+        """nocomma_part : LPAREN any_raw_toks_opt RPAREN
+                        | LBRACE any_raw_toks_opt RBRACE
+                        | LBRACKET any_raw_toks_opt RBRACKET
+                        | AT_LPAREN any_raw_toks_opt RPAREN
+                        | BANG_LPAREN any_raw_toks_opt RPAREN
+                        | BANG_LBRACKET any_raw_toks_opt RBRACKET
+                        | DOLLAR_LPAREN any_raw_toks_opt RPAREN
+                        | DOLLAR_LBRACE any_raw_toks_opt RBRACE
+                        | DOLLAR_LBRACKET any_raw_toks_opt RBRACKET
+                        | ATDOLLAR_LPAREN any_raw_toks_opt RPAREN
+        """
+        pass
+
+    def p_nocomma_base(self, p):
+        """nocomma : nocomma_part"""
+        pass
+
+    def p_nocomma_append(self, p):
+        """nocomma : nocomma nocomma_part"""
+        pass
+
+    def p_comma_nocomma(self, p):
+        """comma_nocomma : comma_tok nocomma"""
+        p1 = p[1]
+        p[0] = [(p1.lineno, p1.lexpos)]
+
+    def p_macroarglist_single(self, p):
+        """macroarglist : nocomma"""
+        p[0] = []
+
+    def p_macroarglist_many(self, p):
+        """macroarglist : nocomma comma_nocomma_list"""
+        p[0] = p[2]
+
     def p_subscriptlist(self, p):
         """subscriptlist : subscript comma_subscript_list_opt comma_opt"""
         p1, p2 = p[1], p[2]
@@ -2340,11 +2480,9 @@ class BaseParser(object):
             else:
                 self._parse_error(p.value,
                                   self.currloc(lineno=p.lineno,
-                                               column=p.lexpos),
-                                  self.xonsh_code)
+                                               column=p.lexpos))
         else:
             msg = 'code: {0}'.format(p.value),
             self._parse_error(msg,
                               self.currloc(lineno=p.lineno,
-                                           column=p.lexpos),
-                              self.xonsh_code)
+                                           column=p.lexpos))