Completion context (#4017)

* completion-context: Add CompletionContextParser placeholder

Implements the xonsh (tab-)completion context parser.
This parser is meant to parse a (possibly incomplete) command line.

* completers: tools: Implement ``contextual_completer`` decorator

This is used to mark completers that want to use the parsed completion context.

* completers: Enable using contextual completers in xonsh/completer.py

* completers: readline, ptk, jupyter: Enable using contextual completers

Pass ``multiline_text`` and ``cursor_index`` to ``Completer.complete()``

* parsers: base: Refactor out a ``raise_parse_error`` function

* tokenize: Enable ``tolerant`` mode

If ``tolerant`` is True, yield ERRORTOKEN instead of
    throwing an exception when encountering an error.

* lexer: Enable ``tolerant`` mode

Tokenize without extra checks (e.g. paren matching).
When True, ERRORTOKEN contains the erroneous string instead of an error msg.

* tests: lexer: Test ``tolerant`` mode

* completion-context: Implement simple CommandContext parsing

* completion-context: tests: Test simple CommandContext parsing

* completion-context: Implement parsing sub-commands

* completion-context: tests: Test parsing sub-commands

* completion-context: Add news file

* completion-context: parser: Add parser table path to relevant locations

Code-coverage, mypy ignore list, etc.

* completion-context: Implement parsing partial strings and line continuations

* completion-context: tests: Test parsing partial strings and line continuations

* completion-context: Convert ``Span`` object to a ``slice``

* completion-context: Refactor out ``create_command`` and ``cursor_in_span``

* completion-context: Implement handling empty commands

* completion-context: tests: Test handling empty commands

* completion-context: Implement handling multiple commands

Separated by newlines, `;`, `and`, `or`, `|`, `&&`, `||`

* completion-context: tests: Test handling multiple commands

Separated by newlines, `;`, `and`, `or`, `|`, `&&`, `||`

* completion-context: Implement handling python context

* completion-context: tests: Test handling python context

* completers: tools: Add `contextual_command_completer`

* completers: Make `complete_skipper` contextual

* completers: Make `complete_from_man` contextual

* completers: Make `complete_from_bash` contextual and add test

* completers: Make `complete_pip` contextual and update tests

* completers: Keep opening string quote if it exists

* completion-context: Handle cursor after a closing quote

For example - cursor at the end of ``ls "/usr/"``.
1. The closing quote will be appended to all completions.
 I.e the completion ``/usr/bin`` will turn into ``/usr/bin"``
2. If not specified, lprefix will cover the closing prefix.
 I.e for ``ls "/usr/"``, the default lprefix will be 6 to include the closing quote.

* completion-context: tests: Test handling cursor after a closing quote

* completion-context: Fix bug with multiple empty commands

e.g. `;;;`

* completion-context: tests: Speed up tests

From ~15 seconds to ~500 ms

* completion-context: Expand commands and subcommands

* completion-context: Simplify `commands` rules

* completion-context: Simplify `sub_expression` rules

* completion-context: Simplify editing a multi-command token

* completion-context: Inline `create_command`

* completion-context: Implement `contextual_command_completer_for` helper

* completers: Make `complete_cd`/`complete_rmdir` contextual and add tests

* completers: path: Don't append a double-backslash in a raw string

When completing a path, if a raw string is used (e.g. `r"C:\Windows\"`),
there's no reason to append a double-backslash (e.g. `r"C:\Windows\\"`).

* completers: Make `complete_xonfig`/`complete_xontrib` contextual and add tests

* completers: Make `complete_completer` contextual and add tests

* completers: Make `complete_import` contextual and add tests

* completion-context: Add python `ctx` attribute

* completion: tools: Simplify `RichCompletion` attributes handling

* completers: Make `base`, `python`, and `commands` contextual

* Add tests
* No need for `python_mode` completer anymore

* completion: tools: Add `append_space` attribute to `RichCompletion`

* completion-context: Get all lines in a main python context

* xontrib: jedi: Make the `jedi` completer contextual

* completers: tools: Remove `get_ptk_completer` and `PromptToolkitCompleter.current_document`

These aren't needed anymore now that contextual completers can access the multiline code (via `PythonContext.multiline_code`).

* completion-context: ptk: Expand aliases

* completion-context: jupyter: Expand aliases and fix line handling

* completer: Preserve custom prefix after closing quote

* completers: bash: Ensure bash completion uses the complete prefix

* completers: pip: Append a space after a pip command

* completers: pip: Prevent bad package name completions

* completers: Remove a common prefix from `RichCompletion` if `display` wasn't provided

* completion-context: Treat cursor at edge of `&& || | ;` as normal args

This will be used for completing a space

* completers: Complete end proc keywords correctly

.coveragerc

@@ -8,6 +8,7 @@ omit =
   */__amalgam__.py
   xonsh/lazyasd.py
   xonsh/parser_table.py
+  xonsh/completion_parser_table.py
   xonsh/ply/*
 
 # keep all cache files in one place

.gitignore

@@ -10,6 +10,8 @@ parser_table.py
 parser_test_table.py
 xonsh/lexer_table.py
 xonsh/parser_table.py
+xonsh/completion_parser_table.py
+xonsh/parsers/completion_parser_table.py
 tests/lexer_table.py
 tests/parser_table.py
 tests/lexer_test_table.py

CONTRIBUTING.rst

@@ -406,7 +406,7 @@ Maintenance Tasks
 You can cleanup your local repository of transient files such as \*.pyc files
 created by unit testing by running::
 
-    $ rm -f xonsh/parser_table.py
+    $ rm -f xonsh/parser_table.py xonsh/completion_parser_table.py
     $ rm -f xonsh/*.pyc tests/*.pyc
     $ rm -fr build
 

news/completion-context.rst

@@ -0,0 +1,23 @@
+**Added:**
+
+* Completion Context - Allow completers to access a parsed representation of the current commandline context.
+
+**Changed:**
+
+* <news item>
+
+**Deprecated:**
+
+* <news item>
+
+**Removed:**
+
+* <news item>
+
+**Fixed:**
+
+* <news item>
+
+**Security:**
+
+* <news item>

setup.cfg

@@ -10,6 +10,7 @@ exclude =
     docs/,
     */ply/,
     parser*_table.py,
+    completion_parser_table.py,
     build/,
     dist/,
     setup.py,
@@ -106,7 +107,7 @@ pretty = True
 
 # the __init__ files have dynamic check - ignoring the attribute error. others are generated files
 # top level package name only ignores the __init__.py file.
-[mypy-xonsh.parser_table,xonsh.parsers.parser_table.*,*.__amalgam__.*,xonsh,xonsh.prompt,xonsh.history,xonsh.completers,xonsh.procs]
+[mypy-xonsh.parser_table,xonsh.completion_parser_table,xonsh.parsers.parser_table.*,xonsh.parsers.completion_parser_table.*,*.__amalgam__.*,xonsh,xonsh.prompt,xonsh.history,xonsh.completers,xonsh.procs]
 ignore_errors = True
 
 # 3rd party libraries that we dont have control over

setup.py

@@ -17,6 +17,7 @@ from setuptools.command.install_scripts import install_scripts
 TABLES = [
     "xonsh/lexer_table.py",
     "xonsh/parser_table.py",
+    "xonsh/completion_parser_table.py",
     "xonsh/__amalgam__.py",
     "xonsh/completers/__amalgam__.py",
     "xonsh/history/__amalgam__.py",

tests/completers/test_base_completer.py

@@ -0,0 +1,43 @@
+import pytest
+
+from tests.tools import ON_WINDOWS
+from xonsh.completers.base import complete_base
+from xonsh.parsers.completion_context import (
+    CompletionContext,
+    CommandContext,
+    CommandArg,
+    PythonContext,
+)
+
+
+CUR_DIR = "." if ON_WINDOWS else "./"  # for some reason this is what happens in `complete_path`
+
+
+@pytest.fixture(autouse=True)
+def setup(xonsh_builtins, xonsh_execer, monkeypatch):
+    monkeypatch.setattr(xonsh_builtins.__xonsh__, "commands_cache", ["cool"])
+
+
+def test_empty_line():
+    completions = complete_base(
+        CompletionContext(
+            command=CommandContext((), 0),
+            python=PythonContext("", 0)
+        )
+    )
+    assert completions
+    for exp in ["cool", CUR_DIR, "abs"]:
+        assert exp in completions
+
+
+def test_empty_subexpr():
+    completions = complete_base(
+        CompletionContext(
+            command=CommandContext((), 0, subcmd_opening="$("),
+            python=None
+        )
+    )
+    assert completions
+    for exp in ["cool", CUR_DIR]:
+        assert exp in completions
+    assert "abs" not in completions

tests/completers/test_bash_completer.py

@@ -0,0 +1,31 @@
+import pytest
+
+from tests.tools import skip_if_on_windows, skip_if_on_darwin
+
+from xonsh.completers.bash import complete_from_bash
+from xonsh.parsers.completion_context import CompletionContext, CommandContext, CommandArg
+
+
+@pytest.fixture(autouse=True)
+def setup(monkeypatch, tmp_path):
+    (tmp_path / "testdir").mkdir()
+    (tmp_path / "spaced dir").mkdir()
+    monkeypatch.chdir(str(tmp_path))
+
+
+
+@skip_if_on_darwin
+@skip_if_on_windows
+@pytest.mark.parametrize("command_context, completions, lprefix", (
+        (CommandContext(args=(CommandArg("bash"),), arg_index=1, prefix="--deb"), {"--debug", "--debugger"}, 5),
+        (CommandContext(args=(CommandArg("ls"),), arg_index=1, prefix=""), {"'testdir/'", "'spaced dir/'"}, 0),
+        (CommandContext(args=(CommandArg("ls"),), arg_index=1, prefix="", opening_quote="'"), {"'testdir/'", "'spaced dir/'"}, 1),
+        pytest.param(CommandContext(args=(CommandArg("ls"),), arg_index=1, prefix="test", opening_quote="'"), {"'testdir/'"}, 1,
+            marks=pytest.mark.skip("bash completions don't consider the opening quote")),
+))
+def test_bash_completer(command_context, completions, lprefix, xonsh_builtins, monkeypatch):
+    if not xonsh_builtins.__xonsh__.env.get("BASH_COMPLETIONS"):
+        monkeypatch.setitem(xonsh_builtins.__xonsh__.env, "BASH_COMPLETIONS", ["/usr/share/bash-completion/bash_completion"])
+    bash_completions, bash_lprefix = complete_from_bash(CompletionContext(command_context))
+    assert bash_completions == completions and bash_lprefix == lprefix
+

tests/completers/test_completer_command.py

@@ -0,0 +1,14 @@
+from xonsh.parsers.completion_context import CommandArg, CommandContext, CompletionContext
+from xonsh.completers.completer import complete_completer
+
+
+def test_options():
+    assert complete_completer(CompletionContext(CommandContext(
+        args=(CommandArg("completer"),), arg_index=1,
+    ))) == {"add", "remove", "list", "help"}
+
+
+def test_help_options():
+    assert complete_completer(CompletionContext(CommandContext(
+        args=(CommandArg("completer"),CommandArg("help")), arg_index=2,
+    ))) == {"add", "remove", "list"}

tests/completers/test_dir_completers.py

@@ -0,0 +1,86 @@
+import re
+import pytest
+import tempfile
+from os import sep
+
+from xonsh.completers.tools import RichCompletion
+from xonsh.completers.dirs import complete_cd, complete_rmdir
+from xonsh.parsers.completion_context import (
+    CompletionContext, CommandContext, CommandArg,
+)
+
+from tests.tools import ON_WINDOWS
+
+COMPLETERS = {
+    "cd": complete_cd,
+    "rmdir": complete_rmdir,
+}
+
+
+@pytest.fixture(autouse=True)
+def setup(xonsh_builtins, xonsh_execer):
+    with tempfile.TemporaryDirectory() as tmp:
+        xonsh_builtins.__xonsh__.env["XONSH_DATA_DIR"] = tmp
+        xonsh_builtins.__xonsh__.env["CDPATH"] = set()
+
+
+@pytest.fixture(params=list(COMPLETERS))
+def cmd(request):
+    return request.param
+
+
+def test_not_cmd(cmd):
+    """Ensure the cd completer doesn't complete other commands"""
+    assert not COMPLETERS[cmd](CompletionContext(CommandContext(
+        args=(CommandArg(f"not-{cmd}"),), arg_index=1,
+    )))
+
+
+def complete_cmd(cmd, prefix, opening_quote="", closing_quote=""):
+    result = COMPLETERS[cmd](CompletionContext(CommandContext(
+            args=(CommandArg(cmd),), arg_index=1, prefix=prefix,
+            opening_quote=opening_quote, closing_quote=closing_quote,
+            is_after_closing_quote=bool(closing_quote),
+        )))
+    assert result and len(result) == 2
+    completions, lprefix = result
+    assert lprefix == len(opening_quote) + len(prefix) + len(closing_quote)  # should override the quotes
+    return completions
+
+
+def test_non_dir(cmd):
+    with tempfile.NamedTemporaryFile(dir=".", suffix="_dummySuffix") as tmp:
+        with pytest.raises(StopIteration):  # tmp is a file
+            completions = complete_cmd(cmd, tmp.name[:-2])
+
+
+@pytest.fixture(scope="module")
+def dir_path():
+    with tempfile.TemporaryDirectory(dir=".", suffix="_dummyDir") as tmp_path:
+        yield tmp_path
+
+
+def test_dirs_only(cmd, dir_path):
+    completions = complete_cmd(cmd, dir_path[:-2])
+    assert completions == {dir_path + sep}
+
+
+def test_opening_quotes(cmd, dir_path):
+    assert complete_cmd(cmd, dir_path, opening_quote="r'") == {f"r'{dir_path}{sep}'"}
+
+
+def test_closing_quotes(cmd, dir_path):
+    prefix = dir_path
+    exp = f"'''{dir_path}{sep}'''"
+    if ON_WINDOWS:
+        prefix = prefix.replace("\\", "\\\\")
+        # the path completer converts to a raw string if there's a backslash
+        exp = "r" + exp
+
+    completions = complete_cmd(cmd, prefix, opening_quote="'''", closing_quote="'''")
+
+    assert completions == {exp}
+
+    completion = completions.pop()
+    assert isinstance(completion, RichCompletion)
+    assert completion.append_closing_quote is False

tests/completers/test_pip_completer.py

@@ -1,30 +1,17 @@
 import pytest
 
-from xonsh.completers.pip import PIP_RE, PIP_LIST_RE
+from xonsh.completers.tools import RichCompletion
+from xonsh.completers.pip import PIP_RE, complete_pip
+from xonsh.parsers.completion_context import CompletionContext, CommandContext, CommandArg
 
 
 @pytest.mark.parametrize(
-    "line", ["pip", "xpip search", "$(pip", "![pip", "$[pip", "!(xpip"]
+    "line", ["pip", "xpip", "/usr/bin/pip3", r"C:\Python\Scripts\pip"]
 )
 def test_pip_re(line):
     assert PIP_RE.search(line)
 
 
-@pytest.mark.parametrize(
-    "line",
-    [
-        "pip show",
-        "xpip uninstall",
-        "$(pip show",
-        "![pip uninstall",
-        "$[pip show",
-        "!(xpip uninstall",
-    ],
-)
-def test_pip_list_re(line):
-    assert PIP_LIST_RE.search(line)
-
-
 @pytest.mark.parametrize(
     "line",
     [
@@ -41,3 +28,23 @@ def test_pip_list_re(line):
 )
 def test_pip_list_re1(line):
     assert PIP_RE.search(line) is None
+
+
+def test_commands():
+    comps = complete_pip(CompletionContext(CommandContext(
+        args=(CommandArg("pip3"),), arg_index=1,
+        prefix="c",
+    )))
+    assert comps.intersection({"cache", "check", "config"})
+    for comp in comps:
+        assert isinstance(comp, RichCompletion)
+        assert comp.append_space
+
+
+def test_package_list():
+    comps = complete_pip(CompletionContext(CommandContext(
+        args=(CommandArg("pip3"), CommandArg("show")), arg_index=2,
+    )))
+    assert "Package" not in comps
+    assert "-----------------------------" not in comps
+    assert "pytest" in comps

tests/completers/test_xompletions.py

@@ -0,0 +1,21 @@
+from xonsh.parsers.completion_context import CommandArg, CommandContext, CompletionContext
+from xonsh.completers.xompletions import complete_xonfig, complete_xontrib
+
+
+def test_xonfig():
+    assert complete_xonfig(CompletionContext(CommandContext(
+        args=(CommandArg("xonfig"),), arg_index=1, prefix="-"
+    ))) == {"-h"}
+
+
+def test_xonfig_colors(monkeypatch):
+    monkeypatch.setattr("xonsh.tools.color_style_names", lambda: ["blue", "brown", "other"])
+    assert complete_xonfig(CompletionContext(CommandContext(
+        args=(CommandArg("xonfig"), CommandArg("colors")), arg_index=2, prefix="b"
+    ))) == {"blue", "brown"}
+
+
+def test_xontrib():
+    assert complete_xontrib(CompletionContext(CommandContext(
+        args=(CommandArg("xontrib"),), arg_index=1, prefix="l"
+    ))) == {"list", "load"}

tests/test_completer.py

@@ -0,0 +1,105 @@
+"""Tests for the base completer's logic (xonsh/completer.py)"""
+
+import pytest
+from xonsh.completers.tools import RichCompletion, contextual_command_completer
+
+from xonsh.completer import Completer
+from xonsh.parsers.completion_context import CommandContext
+
+
+@pytest.fixture(scope="session")
+def completer():
+    return Completer()
+
+
+@pytest.fixture
+def completers_mock(xonsh_builtins, monkeypatch):
+    completers = {}
+    monkeypatch.setattr(xonsh_builtins.__xonsh__, "completers", completers)
+    return completers
+
+
+def test_sanity(completer, completers_mock):
+    # no completions:
+    completers_mock["a"] = lambda *a: None
+    assert completer.complete("", "", 0, 0) == (set(), 0)
+    # simple completion:
+    completers_mock["a"] = lambda *a: {"comp"}
+    assert completer.complete("pre", "", 0, 0) == (("comp",), 3)
+    # multiple completions:
+    completers_mock["a"] = lambda *a: {"comp1", "comp2"}
+    assert completer.complete("pre", "", 0, 0) == (("comp1", "comp2"), 3)
+    # custom lprefix:
+    completers_mock["a"] = lambda *a: ({"comp"}, 2)
+    assert completer.complete("pre", "", 0, 0) == (("comp",), 2)
+    # RichCompletion:
+    completers_mock["a"] = lambda *a: {RichCompletion("comp", prefix_len=5)}
+    assert completer.complete("pre", "", 0, 0) == ((RichCompletion("comp", prefix_len=5),), 3)
+
+
+def test_cursor_after_closing_quote(completer, completers_mock):
+    """See ``Completer.complete`` in ``xonsh/completer.py``"""
+    @contextual_command_completer
+    def comp(context: CommandContext):
+        return {context.prefix + "1", context.prefix + "2"}
+
+    completers_mock["a"] = comp
+
+    assert completer.complete("", "", 0, 0, {}, multiline_text="'test'", cursor_index=6) == (
+        ("test1'", "test2'"), 5
+    )
+
+    assert completer.complete("", "", 0, 0, {}, multiline_text="'''test'''", cursor_index=10) == (
+        ("test1'''", "test2'''"), 7
+    )
+
+
+def test_cursor_after_closing_quote_override(completer, completers_mock):
+    """Test overriding the default values"""
+    @contextual_command_completer
+    def comp(context: CommandContext):
+        return {
+            # replace the closing quote with "a"
+            RichCompletion("a", prefix_len=len(context.closing_quote), append_closing_quote=False),
+            # add text after the closing quote
+            RichCompletion(context.prefix + "_no_quote", append_closing_quote=False),
+            # sanity
+            RichCompletion(context.prefix + "1"),
+        }
+
+    completers_mock["a"] = comp
+
+    assert completer.complete("", "", 0, 0, {}, multiline_text="'test'", cursor_index=6) == (
+        (
+            "a",
+            "test1'",
+            "test_no_quote",
+        ), 5
+    )
+
+    assert completer.complete("", "", 0, 0, {}, multiline_text="'''test'''", cursor_index=10) == (
+        (
+            "a",
+            "test1'''",
+            "test_no_quote",
+        ), 7
+    )
+
+def test_append_space(completer, completers_mock):
+    @contextual_command_completer
+    def comp(context: CommandContext):
+        return {
+            RichCompletion(context.prefix + "a", append_space=True),
+            RichCompletion(context.prefix + " ", append_space=False),  # bad usage
+            RichCompletion(context.prefix + "b", append_space=True, append_closing_quote=False),
+        }
+
+    completers_mock["a"] = comp
+
+    assert completer.complete("", "", 0, 0, {}, multiline_text="'test'", cursor_index=6) == (
+        (
+            "test '",
+            "testa' ",
+            "testb ",
+        ), 5
+    )

tests/test_completion_context.py

@@ -0,0 +1,484 @@
+import itertools
+import typing as tp
+
+import pytest
+
+from xonsh.parsers.completion_context import CommandArg, CommandContext, CompletionContextParser, PythonContext
+
+
+DEBUG = False
+MISSING = object()
+X = "\x00"  # cursor marker
+PARSER: tp.Optional[CompletionContextParser] = None
+
+
+@pytest.fixture(scope="module", autouse=True)
+def parser():
+    global PARSER
+    PARSER = CompletionContextParser(debug=DEBUG)
+    yield
+    PARSER = None
+
+
+def parse(command, inner_index):
+    return PARSER.parse(command, inner_index)
+
+
+def assert_match(commandline, command_context=MISSING, python_context=MISSING, is_main_command=False):
+    if X in commandline:
+        index = commandline.index(X)
+        commandline = commandline.replace(X, "")
+    else:
+        index = len(commandline)
+    context = parse(commandline, index)
+    if context is None:
+        raise SyntaxError("Failed to parse the commandline - set DEBUG = True in this file to see the error")
+    if is_main_command and python_context is MISSING:
+        python_context = PythonContext(commandline, index)
+    if command_context is not MISSING:
+        assert context.command == command_context
+    if python_context is not MISSING:
+        assert context.python == python_context
+
+
+COMMAND_EXAMPLES = (
+    (f"comm{X}", CommandContext(args=(), arg_index=0, prefix="comm")),
+    (f" comm{X}", CommandContext(args=(), arg_index=0, prefix="comm")),
+    (f"comm{X}and", CommandContext(args=(), arg_index=0, prefix="comm", suffix="and")),
+    (f"command {X}", CommandContext(args=(CommandArg("command"),), arg_index=1)),
+    (f"{X} command", CommandContext(args=(CommandArg("command"),), arg_index=0)),
+    (f" command {X}", CommandContext(args=(CommandArg("command"),), arg_index=1)),
+    (f"command --{X}", CommandContext(args=(CommandArg("command"),), arg_index=1, prefix="--")),
+    (f"command a {X}", CommandContext(args=(CommandArg("command"), CommandArg("a")), arg_index=2)),
+    (f"command a b{X}", CommandContext(args=(CommandArg("command"), CommandArg("a")), arg_index=2, prefix="b")),
+    (f"command a   b{X}", CommandContext(args=(CommandArg("command"), CommandArg("a")), arg_index=2, prefix="b")),
+    (f"command {X} a", CommandContext(args=(CommandArg("command"), CommandArg("a")), arg_index=1)),
+    (f"command a {X} b", CommandContext(args=(CommandArg("command"), CommandArg("a"), CommandArg("b")), arg_index=2)),
+    (f"command -{X} b", CommandContext(args=(CommandArg("command"), CommandArg("b")), arg_index=1, prefix="-")),
+    (f"command a {X}b", CommandContext(args=(CommandArg("command"), CommandArg("a")), arg_index=2, suffix="b")),
+    (f"command a{X}b", CommandContext(args=(CommandArg("command"),), arg_index=1, prefix="a", suffix="b")),
+    (f"'comm and' a{X}b", CommandContext(
+        args=(CommandArg("comm and", opening_quote="'", closing_quote="'"),), arg_index=1, prefix="a", suffix="b")),
+)
+
+EMPTY_COMMAND_EXAMPLES = (
+    (f"{X}", CommandContext((), 0)),
+    (f" {X}", CommandContext((), 0)),
+    (f"{X} ", CommandContext((), 0)),
+    (f" {X} ", CommandContext((), 0)),
+)
+
+STRING_ARGS_EXAMPLES = (
+    (f"'comm an{X}d'", CommandContext(
+        args=(), arg_index=0, prefix="comm an", suffix="d", opening_quote="'", closing_quote="'")),
+    (f"'comm and{X}'", CommandContext(
+        args=(), arg_index=0, prefix="comm and", suffix="", opening_quote="'", closing_quote="'")),
+    (f"'comm {X}'", CommandContext(
+        args=(), arg_index=0, prefix="comm ", suffix="", opening_quote="'", closing_quote="'")),
+    (f"\"comm an{X}d\"", CommandContext(
+        args=(), arg_index=0, prefix="comm an", suffix="d", opening_quote="\"", closing_quote="\"")),
+    (f"'''comm an{X}d'''", CommandContext(
+        args=(), arg_index=0, prefix="comm an", suffix="d", opening_quote="'''", closing_quote="'''")),
+    (f"fr'comm an{X}d'", CommandContext(
+        args=(), arg_index=0, prefix="comm an", suffix="d", opening_quote="fr'", closing_quote="'")),
+    (f"'()+{X}'", CommandContext(
+        args=(), arg_index=0, prefix="()+", opening_quote="'", closing_quote="'")),
+    (f"'comm and'{X}", CommandContext(
+        args=(), arg_index=0, prefix="comm and", opening_quote="'", closing_quote="'", is_after_closing_quote=True)),
+    (f"'''comm and'''{X}", CommandContext(
+        args=(), arg_index=0, prefix="comm and",
+        opening_quote="'''", closing_quote="'''", is_after_closing_quote=True)),
+)
+
+COMMAND_EXAMPLES += STRING_ARGS_EXAMPLES
+COMMAND_EXAMPLES += EMPTY_COMMAND_EXAMPLES
+
+
+@pytest.mark.parametrize("commandline, context", COMMAND_EXAMPLES)
+def test_command(commandline, context):
+    assert_match(commandline, context, is_main_command=True)
+
+
+@pytest.mark.parametrize("commandline, context", tuple(
+    (commandline, context) for commandline, context in STRING_ARGS_EXAMPLES
+    if commandline.endswith("'") or commandline.endswith('"')
+))
+def test_partial_string_arg(commandline, context):
+    partial_commandline = commandline.rstrip("\"'")
+    partial_context = context._replace(closing_quote="")
+    assert_match(partial_commandline, partial_context, is_main_command=True)
+
+
+CONT = "\\" "\n"
+
+
+@pytest.mark.parametrize("commandline, context", (
+        # line continuations:
+        (f"echo {CONT}a {X}", CommandContext(args=(CommandArg("echo"), CommandArg("a")), arg_index=2)),
+        (f"echo {CONT}{X}a {CONT} b",
+         CommandContext(args=(CommandArg("echo"), CommandArg("b")), arg_index=1, suffix="a")),
+        (f"echo a{CONT}{X}b",
+         CommandContext(args=(CommandArg("echo"),), arg_index=1, prefix="a", suffix="b")),
+        (f"echo a{X}{CONT}b",
+         CommandContext(args=(CommandArg("echo"),), arg_index=1, prefix="a", suffix="b")),
+        (f"echo ${CONT}(a) {CONT} {X}b",
+         CommandContext(args=(CommandArg("echo"), CommandArg("$(a)")), arg_index=2, suffix="b")),
+
+        # line continuations in strings:
+        (f"echo 'a{CONT}{X}b'",
+         CommandContext(args=(CommandArg("echo"),), arg_index=1, prefix="a", suffix="b", opening_quote="'",
+                        closing_quote="'")),
+        (f"echo '''a{CONT}{X}b'''",
+         CommandContext(args=(CommandArg("echo"),), arg_index=1, prefix="a", suffix="b", opening_quote="'''",
+                        closing_quote="'''")),
+        (f"echo 'a{CONT}{X}b",
+         CommandContext(args=(CommandArg("echo"),), arg_index=1, prefix="a", suffix="b", opening_quote="'")),
+        (f"echo '''a{CONT}{X}b",
+         CommandContext(args=(CommandArg("echo"),), arg_index=1, prefix="a", suffix="b", opening_quote="'''")),
+        (f"echo ''{CONT}'a{X}b",
+         CommandContext(args=(CommandArg("echo"),), arg_index=1, prefix="a", suffix="b", opening_quote="'''")),
+        (f"echo '''a{CONT}{X} b",
+         CommandContext(args=(CommandArg("echo"),), arg_index=1, prefix="a", suffix=" b", opening_quote="'''")),
+
+        # triple-quoted strings:
+        (f"echo '''a\nb{X}\nc'''", CommandContext(
+            args=(CommandArg("echo"),), arg_index=1,
+            prefix="a\nb", suffix="\nc", opening_quote="'''", closing_quote="'''")),
+        (f"echo '''a\n b{X} \n  c'''", CommandContext(
+            args=(CommandArg("echo"),), arg_index=1,
+            prefix="a\n b", suffix=" \n  c", opening_quote="'''", closing_quote="'''")),
+
+        # partial triple-quoted strings:
+        (f"echo '''a\nb{X}\nc", CommandContext(
+            args=(CommandArg("echo"),), arg_index=1,
+            prefix="a\nb", suffix="\nc", opening_quote="'''")),
+        (f"echo '''a\n b{X} \n  c", CommandContext(
+            args=(CommandArg("echo"),), arg_index=1,
+            prefix="a\n b", suffix=" \n  c", opening_quote="'''")),
+))
+def test_multiline_command(commandline, context):
+    assert_match(commandline, context, is_main_command=True)
+
+
+NESTING_EXAMPLES = (
+    # nesting, prefix
+    (f"echo $({X})", "$("),
+    (f"echo @$({X})", "@$("),
+    (f"echo $(echo $({X}))", "$("),
+    (f"echo @(x + $({X}))", "$("),
+    (f"!({X})", "!("),
+    (f"$[{X}]", "$["),
+    (f"![{X}]", "!["),
+)
+
+NESTED_SIMPLE_CMD_EXAMPLES = [
+    (nesting, f"simple {X}", CommandContext(args=(CommandArg("simple"),), arg_index=1, subcmd_opening=prefix))
+    for nesting, prefix in NESTING_EXAMPLES[1:]]
+
+
+@pytest.mark.parametrize("nesting, commandline, context", list(itertools.chain((
+        # complex subcommand in a simple nested expression
+        (NESTING_EXAMPLES[0][0], commandline, context._replace(subcmd_opening=NESTING_EXAMPLES[0][1]))
+        for commandline, context in COMMAND_EXAMPLES
+), NESTED_SIMPLE_CMD_EXAMPLES)))
+def test_nested_command(commandline, context, nesting):
+    nested_commandline = nesting.replace(X, commandline)
+    assert_match(nested_commandline, command_context=context, python_context=None)
+
+
+NESTING_MALFORMATIONS = (
+    lambda s: s[:-1],  # remove the last closing brace ')' / ']'
+    lambda s: s + s[-1],  # add an extra closing brace ')' / ']'
+    lambda s: s[-1] + s,
+    lambda s: s + "$(",
+    lambda s: "$(" + s,
+)
+
+
+@pytest.mark.parametrize("nesting, commandline, context", NESTED_SIMPLE_CMD_EXAMPLES)
+@pytest.mark.parametrize("malformation", NESTING_MALFORMATIONS)
+def test_malformed_subcmd(nesting, commandline, context, malformation):
+    nested_commandline = nesting.replace(X, commandline)
+    nested_commandline = malformation(nested_commandline)
+    assert_match(nested_commandline, command_context=context, python_context=None)
+
+
+MALFORMED_SUBCOMMANDS_NESTINGS = (
+    # nesting, subcmd_opening
+        (f"echo $(a $({X}", "$("),
+        (f"echo $(a $(b; {X}", ""),
+        (f"$(echo $(a $({X}", "$("),
+        (f"echo $[a $({X}]", "$("),
+        (f"echo $(a $[{X})", "$["),
+        (f"echo @(x = $({X}", "$("),
+        (f"echo @(a; x = $({X}", "$("),
+        (f"echo @(x = $(a; {X}", ""),
+)
+
+@pytest.mark.parametrize("nesting, subcmd_opening", MALFORMED_SUBCOMMANDS_NESTINGS)
+@pytest.mark.parametrize("commandline, context", COMMAND_EXAMPLES[:5])
+def test_multiple_malformed_subcmds(nesting, subcmd_opening, commandline, context):
+    nested_commandline = nesting.replace(X, commandline)
+    nested_context = context._replace(subcmd_opening=subcmd_opening)
+    assert_match(nested_commandline, nested_context, python_context=None)
+
+
+def test_other_subcommand_arg():
+    command = "echo $(pwd) "
+    assert_match(
+        command,
+        CommandContext((CommandArg("echo"), CommandArg("$(pwd)")), arg_index=2),
+        is_main_command=True
+    )
+
+
+def test_combined_subcommand_arg():
+    command = f"echo file=$(pwd{X})/x"
+
+    # index inside the subproc
+    assert_match(command, CommandContext(
+        (), arg_index=0, prefix="pwd", subcmd_opening="$("), python_context=None)
+
+    # index at the end of the command
+    assert_match(command.replace(X, ""), CommandContext(
+        (CommandArg("echo"),), arg_index=1, prefix="file=$(pwd)/x"), is_main_command=True)
+
+
+SUBCMD_BORDER_EXAMPLES = (
+    (f"{X}$(echo)", CommandContext((), 0, suffix="$(echo)")),
+    (f"${X}(echo)", CommandContext((), 0, prefix="$", suffix="(echo)")),
+    (f"$(echo){X}", CommandContext((), 0, prefix="$(echo)")),
+    (f"${X}( echo)", CommandContext((), 0, prefix="$", suffix="( echo)")),
+    (f"$(echo ){X}", CommandContext((), 0, prefix="$(echo )")),
+)
+
+
+@pytest.mark.parametrize("commandline, context", SUBCMD_BORDER_EXAMPLES)
+def test_cursor_in_subcmd_borders(commandline, context):
+    assert_match(commandline, context, is_main_command=True)
+
+
+MULTIPLE_COMMAND_KEYWORDS = (
+    "; ",
+    "\n",
+    " and ",
+    "&& ",
+    " or ",
+    "|| ",
+    "| ",
+)
+
+MULTIPLE_CMD_SIMPLE_EXAMPLES = [
+    (keyword, ("echo hi", f"simple {X}"), CommandContext(args=(CommandArg("simple"),), arg_index=1))
+    for keyword in MULTIPLE_COMMAND_KEYWORDS]
+
+EXTENSIVE_COMMAND_PAIRS = tuple(itertools.chain(
+    zip(COMMAND_EXAMPLES, COMMAND_EXAMPLES[::-1]),
+    zip(COMMAND_EXAMPLES, EMPTY_COMMAND_EXAMPLES),
+    zip(EMPTY_COMMAND_EXAMPLES, COMMAND_EXAMPLES),
+    zip(EMPTY_COMMAND_EXAMPLES, EMPTY_COMMAND_EXAMPLES),
+))
+
+MULTIPLE_COMMAND_EXTENSIVE_EXAMPLES = tuple(itertools.chain(
+    (
+        # cursor in first command
+        ((first, second.replace(X, "")), first_context)
+        for (first, first_context), (second, second_context) in EXTENSIVE_COMMAND_PAIRS
+    ),
+    (
+        # cursor in second command
+        ((first.replace(X, ""), second), second_context)
+        for (first, first_context), (second, second_context) in EXTENSIVE_COMMAND_PAIRS
+    ),
+    (
+        # cursor in middle command
+        ((first.replace(X, ""), second, third.replace(X, "")), second_context)
+        for (first, _1), (second, second_context), (third, _3)
+        in zip(COMMAND_EXAMPLES[:3], COMMAND_EXAMPLES[3:6], COMMAND_EXAMPLES[6:9])
+    ),
+    (
+        # cursor in third command
+        ((first.replace(X, ""), second.replace(X, ""), third), third_context)
+        for (first, _1), (second, _2), (third, third_context)
+        in zip(COMMAND_EXAMPLES[:3], COMMAND_EXAMPLES[3:6], COMMAND_EXAMPLES[6:9])
+    ),
+))
+
+
+@pytest.mark.parametrize("keyword, commands, context", tuple(itertools.chain(
+    (
+            (MULTIPLE_COMMAND_KEYWORDS[0], commands, context)
+            for commands, context in MULTIPLE_COMMAND_EXTENSIVE_EXAMPLES
+    ),
+    MULTIPLE_CMD_SIMPLE_EXAMPLES,
+)))
+def test_multiple_commands(keyword, commands, context):
+    joined_command = keyword.join(commands)
+
+    cursor_command = next(command for command in commands if X in command)
+    if cursor_command is commands[0]:
+        relative_index = cursor_command.index(X)
+    else:
+        absolute_index = joined_command.index(X)
+        relative_index = absolute_index - joined_command.rindex(keyword, 0, absolute_index) - len(keyword)
+        if keyword.endswith(" "):
+            # the last space is part of the command
+            relative_index += 1
+            cursor_command = " " + cursor_command
+
+    assert_match(joined_command, context, is_main_command=True)
+
+
+@pytest.mark.parametrize("commandline", (
+    f"{X};",
+    f"; {X}",
+    f"{X};;",
+    f"; {X};",
+    f";; {X}",
+    f";;; {X}",
+))
+def test_multiple_empty_commands(commandline):
+    assert_match(commandline, CommandContext((), 0), is_main_command=True)
+
+
+@pytest.mark.parametrize("nesting, keyword, commands, context", tuple(
+    (nesting, keyword, commands, context)  # no subcmd_opening in nested multi-commands
+    for nesting, prefix in NESTING_EXAMPLES
+    for keyword, commands, context in MULTIPLE_CMD_SIMPLE_EXAMPLES
+    if keyword != "\n"  # the lexer ignores newlines inside subcommands
+))
+def test_nested_multiple_commands(nesting, keyword, commands, context):
+    joined_command = keyword.join(commands)
+    nested_joined = nesting.replace(X, joined_command)
+    assert_match(nested_joined, context, python_context=None)
+
+
+def test_multiple_nested_commands():
+    assert_match(f"echo hi; echo $(ls{X})",
+                 CommandContext((), 0, prefix="ls", subcmd_opening="$("),
+                 python_context=None)
+
+
+@pytest.mark.parametrize("commandline, context", tuple(
+    (commandline, context) for commandline, context in STRING_ARGS_EXAMPLES
+    if commandline.endswith("'") or commandline.endswith('"')
+))
+def test_multiple_partial_string_arg(commandline, context):
+    partial_commandline = commandline.rstrip("\"'")
+    partial_context = context._replace(closing_quote="")
+    assert_match("echo;" + partial_commandline, partial_context)
+    assert_match("echo $[a ;" + partial_commandline, partial_context)
+
+
+@pytest.mark.parametrize("nesting, keyword, commands, context", tuple(
+    (nesting, keyword, commands, context)
+    for nesting, prefix in NESTING_EXAMPLES[:1]
+    for keyword, commands, context in MULTIPLE_CMD_SIMPLE_EXAMPLES[:1]
+))
+@pytest.mark.parametrize("malformation", NESTING_MALFORMATIONS)
+def test_malformed_subcmd(malformation, nesting, keyword, commands, context):
+    joined_command = keyword.join(commands)
+    nested_joined = nesting.replace(X, joined_command)
+    malformed_commandline = malformation(nested_joined)
+    assert_match(malformed_commandline, context, python_context=None)
+
+
+MULTIPLE_COMMAND_BORDER_EXAMPLES = tuple(itertools.chain(
+    itertools.chain(*(
+        (
+            (f"ls{ws1}{X}{kwd}{ws2}echo",
+             CommandContext((CommandArg("ls"),), 1) if ws1 else CommandContext((), 0, prefix="ls")),
+        ) for ws1, ws2, kwd in itertools.product(("", " "), ("", " "), ("&&", ";"))
+    )),
+
+    # all keywords are treated as a normal arg if the cursor is at the edge
+    (
+        (f"ls {X}and echo", CommandContext((CommandArg("ls"), CommandArg("echo")), 1, suffix="and")),
+        (f"ls and{X} echo", CommandContext((CommandArg("ls"), CommandArg("echo")), 1, prefix="and")),
+        (f"ls ||{X} echo", CommandContext((CommandArg("ls"), CommandArg("echo")), 1, prefix="||")),
+    ),
+
+    # if the cursor is inside the keyword, it's treated as a normal arg
+    (
+        (f"ls a{X}nd echo", CommandContext((CommandArg("ls"), CommandArg("echo")), 1, prefix="a", suffix="nd")),
+        (f"ls &{X}& echo", CommandContext((CommandArg("ls"), CommandArg("echo")), 1, prefix="&", suffix="&")),
+    )
+))
+
+
+@pytest.mark.parametrize("commandline, context", tuple(itertools.chain(
+    MULTIPLE_COMMAND_BORDER_EXAMPLES,
+    (
+            # ensure these rules work with more than one command
+            (f"cat | {commandline}", context)
+            for commandline, context in MULTIPLE_COMMAND_BORDER_EXAMPLES
+    ),
+)))
+def test_cursor_in_multiple_keyword_borders(commandline, context):
+    assert_match(commandline, context)
+
+
+PYTHON_EXAMPLES = (
+    # commandline, context
+    (f"x = {X}", PythonContext("x = ", 4)),
+    (f"a {X}= x; b = y", PythonContext("a = x; b = y", 2)),
+    (f"a {X}= x\nb = $(ls)", PythonContext("a = x\nb = $(ls)", 2)),
+)
+
+PYTHON_NESTING_EXAMPLES = (
+    # nesting, prefix
+    f"echo @({X})",
+    f"echo $(echo @({X}))",
+    f"echo @(x + @({X}))",  # invalid syntax, but can still be in a partial command
+)
+
+
+@pytest.mark.parametrize("nesting, commandline, context", list(itertools.chain((
+        # complex subcommand in a simple nested expression
+        (nesting, commandline, context._replace(is_sub_expression=True))
+        for nesting in PYTHON_NESTING_EXAMPLES[:1]
+        for commandline, context in PYTHON_EXAMPLES
+), (
+        # simple subcommand in a complex nested expression
+        (nesting, commandline, context._replace(is_sub_expression=True))
+        for nesting in PYTHON_NESTING_EXAMPLES
+        for commandline, context in PYTHON_EXAMPLES[:1]
+))))
+def test_nested_python(commandline, context, nesting):
+    nested_commandline = nesting.replace(X, commandline)
+    assert_match(nested_commandline, command_context=None, python_context=context)
+
+
+@pytest.mark.parametrize("commandline, context", [
+    (commandline.replace("$", "@"), context._replace(
+        prefix=context.prefix.replace("$", "@"),
+        suffix=context.suffix.replace("$", "@"),
+    ))
+    for commandline, context in SUBCMD_BORDER_EXAMPLES
+])
+def test_cursor_in_sub_python_borders(commandline, context):
+    assert_match(commandline, context, is_main_command=True)
+
+
+@pytest.mark.parametrize("code", (
+    f"""
+x = 3
+x.{X}""",
+    f"""
+x = 3;
+y = 4;
+x.{X}""",
+    f"""
+def func({X}):
+    return 100
+    """,
+    f"""
+class A:
+    def a():
+        return "a{X}"
+    pass
+exit()
+    """,
+))
+def test_multiline_python(code):
+    assert_match(code, is_main_command=True)

tests/test_jupyter_kernel.py

@@ -0,0 +1,44 @@
+import sys
+import pytest
+from inspect import signature
+from unittest.mock import MagicMock
+
+from xonsh.aliases import Aliases
+from xonsh.completer import Completer
+from tests.test_ptk_completer import EXPANSION_CASES
+
+XonshKernel = None
+
+@pytest.fixture(autouse=True)
+def setup(monkeypatch):
+    global XonshKernel
+    if XonshKernel is None:
+        monkeypatch.setitem(sys.modules, "zmq", MagicMock())
+        monkeypatch.setitem(sys.modules, "zmq.eventloop", MagicMock())
+        monkeypatch.setitem(sys.modules, "zmq.error", MagicMock())
+        import xonsh.jupyter_kernel
+        XonshKernel = xonsh.jupyter_kernel.XonshKernel
+
+
+@pytest.mark.parametrize(
+    "code, index, expected_args",
+    EXPANSION_CASES
+)
+def test_completion_alias_expansion(
+    code, index, expected_args, monkeypatch, xonsh_builtins,
+):
+    xonsh_completer_mock = MagicMock(spec=Completer)
+    xonsh_completer_mock.complete.return_value = set(), 0
+
+    kernel = MagicMock()
+    kernel.completer = xonsh_completer_mock
+
+    monkeypatch.setattr("builtins.aliases", Aliases(gb=["git branch"]))
+    monkeypatch.setattr(xonsh_builtins.__xonsh__.shell, "ctx", None, raising=False)
+
+    XonshKernel.do_complete(kernel, code, index)
+    mock_call = xonsh_completer_mock.complete.call_args
+    args, kwargs = mock_call
+    expected_args["self"] = None
+    expected_args["ctx"] = None
+    assert signature(Completer.complete).bind(None, *args, **kwargs).arguments == expected_args

tests/test_lexer.py

@@ -460,3 +460,22 @@ def test_lexer_split(s, exp):
     lexer = Lexer()
     obs = lexer.split(s)
     assert exp == obs
+
+
+@pytest.mark.parametrize("s", (
+    "()",  # sanity
+    "(",
+    ")",
+    "))",
+    "'string\nliteral",
+    "'''string\nliteral",
+    "string\nliteral'",
+    "\"",
+    "'",
+    "\"\"\"",
+))
+def test_tolerant_lexer(s):
+    lexer = Lexer(tolerant=True)
+    lexer.input(s)
+    error_tokens = list(tok for tok in lexer if tok.type == "ERRORTOKEN")
+    assert all(tok.value in s for tok in error_tokens)  # no error messages

tests/test_man.py

@@ -5,6 +5,8 @@ from xonsh.completers.man import complete_from_man
 
 from tools import skip_if_on_windows
 
+from xonsh.parsers.completion_context import CompletionContext, CommandContext, CommandArg
+
 
 @skip_if_on_windows
 def test_man_completion(monkeypatch, tmpdir, xonsh_builtins):
@@ -13,6 +15,8 @@ def test_man_completion(monkeypatch, tmpdir, xonsh_builtins):
         os.environ, "MANPATH", os.path.dirname(os.path.abspath(__file__))
     )
     xonsh_builtins.__xonsh__.env.update({"XONSH_DATA_DIR": str(tempdir)})
-    completions = complete_from_man("--", "yes --", 4, 6, xonsh_builtins.__xonsh__.env)
+    completions = complete_from_man(CompletionContext(
+        CommandContext(args=(CommandArg("yes"),), arg_index=1, prefix="--")
+    ))
     assert "--version" in completions
     assert "--help" in completions

tests/test_ptk_completer.py

@@ -1,8 +1,11 @@
 import pytest
+from inspect import signature
 from unittest.mock import MagicMock
+from prompt_toolkit.document import Document
 from prompt_toolkit.completion import Completion as PTKCompletion
 
 from xonsh.aliases import Aliases
+from xonsh.completer import Completer
 from xonsh.completers.tools import RichCompletion
 from xonsh.ptk_shell.completer import PromptToolkitCompleter
 
@@ -50,3 +53,140 @@ def test_rich_completion(
         ]
     else:
         assert completions == [ptk_completion]
+
+
+EXPANSION_CASES = (
+    (
+        "sanity", 6,
+        dict(
+            prefix="sanity",
+            line="sanity",
+            begidx=0,
+            endidx=6,
+            multiline_text="sanity",
+            cursor_index=6,
+        ),
+    ),
+    (
+        "gb ", 3,
+        dict(
+            prefix="",
+            line="git branch ",
+            begidx=11,
+            endidx=11,
+            multiline_text="git branch ",
+            cursor_index=11,
+        ),
+    ),
+    (
+        "gb ", 1,
+        dict(
+            prefix="g",
+            line="gb ",
+            begidx=0,
+            endidx=1,
+            multiline_text="gb ",
+            cursor_index=1,
+        ),
+    ),
+    (
+        "gb", 0,
+        dict(
+            prefix="",
+            line="gb",
+            begidx=0,
+            endidx=0,
+            multiline_text="gb",
+            cursor_index=0,
+        ),
+    ),
+    (
+        " gb ", 0,
+        dict(
+            prefix="",
+            line=" gb ",  # the PTK completer `lstrip`s the line
+            begidx=0,
+            endidx=0,
+            multiline_text=" gb ",
+            cursor_index=0,
+        ),
+    ),
+    (
+        "gb --", 5,
+        dict(
+            prefix="--",
+            line="git branch --",
+            begidx=11,
+            endidx=13,
+            multiline_text="git branch --",
+            cursor_index=13,
+        ),
+    ),
+    (
+        "nice\ngb --", 10,
+        dict(
+            prefix="--",
+            line="git branch --",
+            begidx=11,
+            endidx=13,
+            multiline_text="nice\ngit branch --",
+            cursor_index=18,
+        ),
+    ),
+    (
+        "nice\n gb --", 11,
+        dict(
+            prefix="--",
+            line=" git branch --",
+            begidx=12,
+            endidx=14,
+            multiline_text="nice\n git branch --",
+            cursor_index=19,
+        ),
+    ),
+    (
+        "gb -- wow", 5,
+        dict(
+            prefix="--",
+            line="git branch -- wow",
+            begidx=11,
+            endidx=13,
+            multiline_text="git branch -- wow",
+            cursor_index=13,
+        ),
+    ),
+    (
+        "gb --wow", 5,
+        dict(
+            prefix="--",
+            line="git branch --wow",
+            begidx=11,
+            endidx=13,
+            multiline_text="git branch --wow",
+            cursor_index=13,
+        ),
+    ),
+)
+
+@pytest.mark.parametrize(
+    "code, index, expected_args",
+    EXPANSION_CASES
+)
+def test_alias_expansion(
+    code, index, expected_args, monkeypatch, xonsh_builtins
+):
+    xonsh_completer_mock = MagicMock(spec=Completer)
+    xonsh_completer_mock.complete.return_value = set(), 0
+
+    ptk_completer = PromptToolkitCompleter(xonsh_completer_mock, None, None)
+    ptk_completer.reserve_space = lambda: None
+    ptk_completer.suggestion_completion = lambda _, __: None
+
+    monkeypatch.setattr("builtins.aliases", Aliases(gb=["git branch"]))
+
+    list(ptk_completer.get_completions(Document(code, index), MagicMock()))
+    mock_call = xonsh_completer_mock.complete.call_args
+    args, kwargs = mock_call
+    expected_args["self"] = None
+    expected_args["ctx"] = None
+    assert signature(Completer.complete).bind(None, *args, **kwargs).arguments == expected_args

tests/test_python_completers.py

@@ -1,10 +1,13 @@
 import pytest
 
-from xonsh.completers.python import python_signature_complete
+from tests.tools import skip_if_pre_3_8
+from xonsh.completers.python import python_signature_complete, complete_import, complete_python
+from xonsh.parsers.completion_context import CommandArg, CommandContext, CompletionContext, CompletionContextParser, PythonContext
 
 
 @pytest.fixture(autouse=True)
-def xonsh_execer_autouse(xonsh_builtins, xonsh_execer):
+def xonsh_execer_autouse(xonsh_builtins, xonsh_execer, monkeypatch):
+    monkeypatch.setitem(xonsh_builtins.__xonsh__.env, "COMPLETIONS_BRACKETS", True)
     return xonsh_execer
 
 
@@ -50,3 +53,48 @@ def test_complete_python_signatures(line, end, exp):
     ctx = dict(BASE_CTX)
     obs = python_signature_complete("", line, end, ctx, always_true)
     assert exp == obs
+
+
+@pytest.mark.parametrize("code, exp", (
+    ("x = su", "sum"),
+    ("imp", "import"),
+    ("{}.g", "{}.get("),
+    # no signature for native builtins under 3.7:
+    pytest.param("''.split(ma", "maxsplit=", marks=skip_if_pre_3_8),
+))
+def test_complete_python(code, exp):
+    res = complete_python(CompletionContext(python=PythonContext(code, len(code), ctx={})))
+    assert res and len(res) == 2
+    comps, _ = res
+    assert exp in comps
+
+
+def test_complete_python_ctx():
+    class A:
+        def wow():
+            pass
+    
+    a = A()
+
+    res = complete_python(CompletionContext(python=PythonContext("a.w", 2, ctx=locals())))
+    assert res and len(res) == 2
+    comps, _ = res
+    assert "a.wow(" in comps
+
+
+@pytest.mark.parametrize("command, exp", (
+    (CommandContext(args=(CommandArg("import"),), arg_index=1, prefix="pathli"), {"pathlib"}),
+    (CommandContext(args=(CommandArg("from"),), arg_index=1, prefix="pathli"), {"pathlib "}),
+    (CommandContext(args=(CommandArg("import"),), arg_index=1, prefix="os.pa"), {"os.path"}),
+    (CommandContext(args=(
+        CommandArg("import"), CommandArg("os,"),
+        ), arg_index=2, prefix="pathli"), {"pathlib"}),
+    (CommandContext(args=(
+        CommandArg("from"), CommandArg("pathlib"), CommandArg("import"),
+        ), arg_index=3, prefix="PurePa"), {"PurePath"}),
+))
+def test_complete_import(command, exp):
+    result = complete_import(CompletionContext(command,
+        python=PythonContext("", 0)  # `complete_import` needs this
+    ))
+    assert result == exp

tests/xontribs/test_jedi.py

@@ -7,6 +7,7 @@ from unittest.mock import MagicMock, call
 
 from xonsh.xontribs import find_xontrib
 from xonsh.completers.tools import RichCompletion
+from xonsh.parsers.completion_context import CompletionContext, PythonContext
 
 
 @pytest.fixture
@@ -47,16 +48,19 @@ def test_completer_added(jedi_xontrib):
 
 
 @pytest.mark.parametrize(
-    "prefix, line, start, end, ctx", [("x", "10 + x", 5, 6, {}),], ids="x"
+    "context",
+    [
+        CompletionContext(python=PythonContext("10 + x", 6)),
+    ],
 )
 @pytest.mark.parametrize("version", ["new", "old"])
-def test_jedi_api(jedi_xontrib, jedi_mock, version, prefix, line, start, end, ctx):
+def test_jedi_api(jedi_xontrib, jedi_mock, version, context):
     if version == "old":
         jedi_mock.__version__ = "0.15.0"
         jedi_mock.Interpreter().completions.return_value = []
         jedi_mock.reset_mock()
 
-    jedi_xontrib.complete_jedi(prefix, line, start, end, ctx)
+    jedi_xontrib.complete_jedi(context)
 
     extra_namespace = {"__xonsh__": builtins.__xonsh__}
     try:
@@ -65,6 +69,8 @@ def test_jedi_api(jedi_xontrib, jedi_mock, version, prefix, line, start, end, ct
         pass
     namespaces = [{}, extra_namespace]
 
+    line = context.python.multiline_code
+    end = context.python.cursor_index
     if version == "new":
         assert jedi_mock.Interpreter.call_args_list == [call(line, namespaces)]
         assert jedi_mock.Interpreter().complete.call_args_list == [call(1, end)]
@@ -76,14 +82,12 @@ def test_jedi_api(jedi_xontrib, jedi_mock, version, prefix, line, start, end, ct
 
 
 def test_multiline(jedi_xontrib, jedi_mock, monkeypatch):
-    shell_mock = MagicMock()
     complete_document = "xx = 1\n1 + x"
-    shell_mock.shell_type = "prompt_toolkit"
-    shell_mock.shell.pt_completer.current_document.text = complete_document
-    shell_mock.shell.pt_completer.current_document.cursor_position_row = 1
-    shell_mock.shell.pt_completer.current_document.cursor_position_col = 5
-    monkeypatch.setattr(builtins.__xonsh__, "shell", shell_mock)
-    jedi_xontrib.complete_jedi("x", "x", 0, 1, {})
+    jedi_xontrib.complete_jedi(
+        CompletionContext(
+            python=PythonContext(complete_document, len(complete_document))
+        )
+    )
 
     assert jedi_mock.Interpreter.call_args_list[0][0][0] == complete_document
     assert jedi_mock.Interpreter().complete.call_args_list == [
@@ -200,7 +204,9 @@ def test_rich_completions(jedi_xontrib, jedi_mock, completion, rich_completion):
 
     jedi_xontrib.XONSH_SPECIAL_TOKENS = []
     jedi_mock.Interpreter().complete.return_value = [comp_mock]
-    completions = jedi_xontrib.complete_jedi("", "", 0, 0, {})
+    completions = jedi_xontrib.complete_jedi(
+        CompletionContext(python=PythonContext("", 0))
+    )
     assert len(completions) == 1
     (ret_completion,) = completions
     assert isinstance(ret_completion, RichCompletion)
@@ -210,9 +216,7 @@ def test_rich_completions(jedi_xontrib, jedi_mock, completion, rich_completion):
 
 
 def test_special_tokens(jedi_xontrib):
-    assert (
-        jedi_xontrib.complete_jedi("", "", 0, 0, {})
-        == jedi_xontrib.XONSH_SPECIAL_TOKENS
-    )
-    assert jedi_xontrib.complete_jedi("@", "@", 0, 1, {}) == {"@", "@(", "@$("}
-    assert jedi_xontrib.complete_jedi("$", "$", 0, 1, {}) == {"$[", "${", "$("}
+    assert jedi_xontrib.complete_jedi(CompletionContext(python=PythonContext("", 0))) \
+        .issuperset(jedi_xontrib.XONSH_SPECIAL_TOKENS)
+    assert jedi_xontrib.complete_jedi(CompletionContext(python=PythonContext( "@", 1))) == {"@", "@(", "@$("}
+    assert jedi_xontrib.complete_jedi(CompletionContext(python=PythonContext("$", 1))) == {"$[", "${", "$("}

xonsh/__init__.py

@@ -1,7 +1,7 @@
 __version__ = "0.9.27"
 
 
-# amalgamate exclude jupyter_kernel parser_table parser_test_table pyghooks
+# amalgamate exclude jupyter_kernel parser_table completion_parser_table parser_test_table pyghooks
 # amalgamate exclude winutils wizard pytest_plugin fs macutils pygments_cache
 # amalgamate exclude jupyter_shell proc
 import os as _os

xonsh/aliases.py

@@ -108,15 +108,19 @@ class Aliases(cabc.MutableMapping):
                 acc_args = rest + list(acc_args)
                 return self.eval_alias(self._raw[token], seen_tokens, acc_args)
 
-    def expand_alias(self, line):
+    def expand_alias(self, line: str, cursor_index: int) -> str:
         """Expands any aliases present in line if alias does not point to a
         builtin function and if alias is only a single command.
+        The command won't be expanded if the cursor's inside/behind it.
         """
-        word = line.split(" ", 1)[0]
-        if word in builtins.aliases and isinstance(self.get(word), cabc.Sequence):
+        word = (line.split(maxsplit=1) or [""])[0]
+        if word in builtins.aliases and isinstance(self.get(word), cabc.Sequence):  # type: ignore
             word_idx = line.find(word)
-            expansion = " ".join(self.get(word))
-            line = line[:word_idx] + expansion + line[word_idx + len(word) :]
+            word_edge = word_idx + len(word)
+            if cursor_index > word_edge:
+                # the cursor isn't inside/behind the word
+                expansion = " ".join(self.get(word))
+                line = line[:word_idx] + expansion + line[word_edge:]
         return line
 
     #

xonsh/completer.py

@@ -1,15 +1,30 @@
 # -*- coding: utf-8 -*-
 """A (tab-)completer for xonsh."""
 import builtins
+import typing as tp
 import collections.abc as cabc
 
+from xonsh.completers.tools import is_contextual_completer, Completion, RichCompletion
+from xonsh.parsers.completion_context import CompletionContext, CompletionContextParser
 from xonsh.tools import print_exception
 
 
 class Completer(object):
     """This provides a list of optional completions for the xonsh shell."""
 
-    def complete(self, prefix, line, begidx, endidx, ctx=None):
+    def __init__(self):
+        self.context_parser = CompletionContextParser()
+
+    def complete(
+        self,
+        prefix,
+        line,
+        begidx,
+        endidx,
+        ctx=None,
+        multiline_text=None,
+        cursor_index=None,
+    ):
         """Complete the string, given a possible execution context.
 
         Parameters
@@ -22,8 +37,14 @@ class Completer(object):
             The index in line that prefix starts on.
         endidx : int
             The index in line that prefix ends on.
-        ctx : Iterable of str (ie dict, set, etc), optional
+        ctx : dict, optional
             Names in the current execution context.
+        multiline_text : str
+            The complete multiline text. Needed to get completion context.
+        cursor_index : int
+            The current cursor's index in the multiline text.
+            May be ``len(multiline_text)`` for cursor at the end.
+            Needed to get completion context.
 
         Returns
         -------
@@ -32,10 +53,27 @@ class Completer(object):
         lprefix : int
             Length of the prefix to be replaced in the completion.
         """
+
+        if multiline_text is not None and cursor_index is not None:
+            completion_context: tp.Optional[
+                CompletionContext
+            ] = self.context_parser.parse(
+                multiline_text,
+                cursor_index,
+                ctx,
+            )
+        else:
+            completion_context = None
+
         ctx = ctx or {}
         for func in builtins.__xonsh__.completers.values():
             try:
-                out = func(prefix, line, begidx, endidx, ctx)
+                if is_contextual_completer(func):
+                    if completion_context is None:
+                        continue
+                    out = func(completion_context)
+                else:
+                    out = func(prefix, line, begidx, endidx, ctx)
             except StopIteration:
                 return set(), len(prefix)
             except Exception as e:
@@ -45,12 +83,59 @@ class Completer(object):
                     f"{e}"
                 )
                 return set(), len(prefix)
+
+            completing_contextual_command = (
+                is_contextual_completer(func)
+                and completion_context is not None
+                and completion_context.command is not None
+            )
             if isinstance(out, cabc.Sequence):
                 res, lprefix = out
+                custom_lprefix = True
             else:
                 res = out
-                lprefix = len(prefix)
+                custom_lprefix = False
+                if completing_contextual_command:
+                    lprefix = len(completion_context.command.prefix)
+                else:
+                    lprefix = len(prefix)
+
             if res is not None and len(res) != 0:
+                if (
+                    completing_contextual_command
+                    and completion_context.command.is_after_closing_quote
+                ):
+                    """
+                    The cursor is appending to a closed string literal, i.e. cursor at the end of ``ls "/usr/"``.
+                    1. The closing quote will be appended to all completions.
+                       I.e the completion ``/usr/bin`` will turn into ``/usr/bin"``
+                       To prevent this behavior, a completer can return a ``RichCompletion`` with ``append_closing_quote=False``.
+                    2. If not specified, lprefix will cover the closing prefix.
+                       I.e for ``ls "/usr/"``, the default lprefix will be 6 to include the closing quote.
+                       To prevent this behavior, a completer can return a different lprefix or specify it inside ``RichCompletion``.
+                    """
+                    closing_quote = completion_context.command.closing_quote
+                    if not custom_lprefix:
+                        lprefix += len(closing_quote)
+
+                    def append_closing_quote(completion: Completion):
+                        if isinstance(completion, RichCompletion):
+                            if completion.append_closing_quote:
+                                return completion.replace(
+                                    value=completion.value + closing_quote
+                                )
+                            return completion
+                        return completion + closing_quote
+
+                    res = map(append_closing_quote, res)
+
+                # append spaces AFTER appending closing quote
+                def append_space(comp: Completion):
+                    if isinstance(comp, RichCompletion) and comp.append_space:
+                        return comp.replace(value=comp.value + " ")
+                    return comp
+
+                res = map(append_space, res)
 
                 def sortkey(s):
                     return s.lstrip(''''"''').lower()

xonsh/completers/base.py

@@ -1,35 +1,56 @@
 """Base completer for xonsh."""
+import typing as tp
 import collections.abc as cabc
+from xonsh.parsers.completion_context import CompletionContext
+from xonsh.completers.tools import contextual_completer, RichCompletion, Completion
 
-from xonsh.completers.path import complete_path
+from xonsh.completers.path import contextual_complete_path
 from xonsh.completers.python import complete_python
 from xonsh.completers.commands import complete_command
 
 
-def complete_base(prefix, line, start, end, ctx):
+@contextual_completer
+def complete_base(context: CompletionContext):
     """If the line is empty, complete based on valid commands, python names,
     and paths.  If we are completing the first argument, complete based on
     valid commands and python names.
     """
-    if line.strip() and prefix != line:
+    out: tp.Set[Completion] = set()
+    if context.command is None or context.command.arg_index != 0:
         # don't do unnecessary completions
-        return set()
+        return out
 
     # get and unpack python completions
-    python_comps = complete_python(prefix, line, start, end, ctx)
+    python_comps = complete_python(context) or set()
     if isinstance(python_comps, cabc.Sequence):
-        python_comps, python_comps_len = python_comps
+        python_comps, python_comps_len = python_comps  # type: ignore
+        out.update(apply_lprefix(python_comps, python_comps_len))
     else:
-        python_comps_len = None
+        out.update(python_comps)
+
     # add command completions
-    out = python_comps | complete_command(prefix, line, start, end, ctx)
+    out.update(complete_command(context.command))
+
     # add paths, if needed
-    if line.strip() == "":
-        paths = complete_path(prefix, line, start, end, ctx, False)
-        return (out | paths[0]), paths[1]
-    elif prefix == line:
-        if python_comps_len is None:
-            return out
+    if not context.command.prefix:
+        path_comps, path_comp_len = contextual_complete_path(
+            context.command, cdpath=False
+        )
+        out.update(apply_lprefix(path_comps, path_comp_len))
+
+    return out
+
+
+def apply_lprefix(comps, lprefix):
+    if lprefix is None:
+        return comps
+
+    for comp in comps:
+        if isinstance(comp, RichCompletion):
+            if comp.prefix_len is None:
+                yield comp.replace(prefix_len=lprefix)
+            else:
+                # this comp has a custom prefix len
+                yield comp
         else:
-            return out, python_comps_len
-    return set()
+            yield RichCompletion(comp, prefix_len=lprefix)

xonsh/completers/bash.py

@@ -4,14 +4,29 @@ import builtins
 import xonsh.platform as xp
 from xonsh.completers.path import _quote_paths
 from xonsh.completers.bash_completion import bash_completions
+from xonsh.completers.tools import contextual_command_completer, RichCompletion
+from xonsh.parsers.completion_context import CommandContext
 
 
-def complete_from_bash(prefix, line, begidx, endidx, ctx):
+@contextual_command_completer
+def complete_from_bash(context: CommandContext):
     """Completes based on results from BASH completion."""
-    env = builtins.__xonsh__.env.detype()
-    paths = builtins.__xonsh__.env.get("BASH_COMPLETIONS", ())
+    env = builtins.__xonsh__.env.detype()  # type: ignore
+    paths = builtins.__xonsh__.env.get("BASH_COMPLETIONS", ())  # type: ignore
     command = xp.bash_command()
-    return bash_completions(
+    # TODO: Allow passing the parsed data directly to py-bash-completion
+    args = [arg.raw_value for arg in context.args]
+    prefix = context.raw_prefix
+    args.insert(context.arg_index, prefix)
+    line = " ".join(args)
+
+    # lengths of all args + joining spaces
+    begidx = sum(len(a) for a in args[: context.arg_index]) + max(
+        context.arg_index - 1, 0
+    )
+    endidx = begidx + len(prefix)
+
+    comps, lprefix = bash_completions(
         prefix,
         line,
         begidx,
@@ -21,3 +36,11 @@ def complete_from_bash(prefix, line, begidx, endidx, ctx):
         command=command,
         quote_paths=_quote_paths,
     )
+
+    def handle_space(comp: str):
+        if comp.endswith(" "):
+            return RichCompletion(comp[:-1], append_space=True)
+        return comp
+
+    comps = set(map(handle_space, comps))
+    return comps, lprefix

xonsh/completers/commands.py

@@ -1,23 +1,31 @@
 import os
 import builtins
+import typing as tp
 
 import xonsh.tools as xt
 import xonsh.platform as xp
-
-from xonsh.completers.tools import get_filter_function
+from xonsh.completers.tools import (
+    get_filter_function,
+    contextual_command_completer,
+    is_contextual_completer,
+    RichCompletion,
+    Completion,
+)
+from xonsh.parsers.completion_context import CompletionContext, CommandContext
 
 SKIP_TOKENS = {"sudo", "time", "timeit", "which", "showcmd", "man"}
-END_PROC_TOKENS = {"|", "||", "&&", "and", "or"}
+END_PROC_TOKENS = ("|", ";", "&&")  # includes ||
+END_PROC_KEYWORDS = {"and", "or"}
 
 
-def complete_command(cmd, line, start, end, ctx):
+def complete_command(command: CommandContext):
     """
     Returns a list of valid commands starting with the first argument
     """
-    space = " "
-    out = {
-        s + space
-        for s in builtins.__xonsh__.commands_cache
+    cmd = command.prefix
+    out: tp.Set[Completion] = {
+        RichCompletion(s, append_space=True)
+        for s in builtins.__xonsh__.commands_cache  # type: ignore
         if get_filter_function()(s, cmd)
     }
     if xp.ON_WINDOWS:
@@ -30,35 +38,59 @@ def complete_command(cmd, line, start, end, ctx):
     return out
 
 
-def complete_skipper(cmd, line, start, end, ctx):
+@contextual_command_completer
+def complete_skipper(command_context: CommandContext):
     """
-    Skip over several tokens (e.g., sudo) and complete based on the rest of the
-    line.
+    Skip over several tokens (e.g., sudo) and complete based on the rest of the command.
+
+    Contextual completers don't need us to skip tokens since they get the correct completion context -
+    meaning we only need to skip commands like ``sudo``.
     """
-    parts = line.split(" ")
     skip_part_num = 0
-    for i, s in enumerate(parts):
-        if s in END_PROC_TOKENS:
-            skip_part_num = i + 1
-    while len(parts) > skip_part_num:
-        if parts[skip_part_num] not in SKIP_TOKENS:
+    for skip_part_num, arg in enumerate(
+        command_context.args[: command_context.arg_index]
+    ):
+        # all the args before the current argument
+        if arg.value not in SKIP_TOKENS:
             break
-        skip_part_num += 1
 
     if skip_part_num == 0:
-        return set()
+        return None
 
-    # If there's no space following an END_PROC_TOKEN, insert one
-    if parts[-1] in END_PROC_TOKENS:
-        return (set(" "), 0)
-
-    if len(parts) == skip_part_num + 1:
-        comp_func = complete_command
-    else:
-        comp = builtins.__xonsh__.shell.shell.completer
-        comp_func = comp.complete
-
-    skip_len = len(" ".join(line[:skip_part_num])) + 1
-    return comp_func(
-        cmd, " ".join(parts[skip_part_num:]), start - skip_len, end - skip_len, ctx
+    skipped_context = CompletionContext(
+        command=command_context._replace(
+            args=command_context.args[skip_part_num:],
+            arg_index=command_context.arg_index - skip_part_num,
+        )
     )
+
+    completers = builtins.__xonsh__.completers.values()  # type: ignore
+    for completer in completers:
+        if is_contextual_completer(completer):
+            results = completer(skipped_context)
+            if results:
+                return results
+    return None
+
+
+@contextual_command_completer
+def complete_end_proc_tokens(command_context: CommandContext):
+    """If there's no space following an END_PROC_TOKEN, insert one"""
+    if command_context.opening_quote or not command_context.prefix:
+        return None
+    prefix = command_context.prefix
+    # for example `echo a|`, `echo a&&`, `echo a ;`
+    if any(prefix.endswith(ending) for ending in END_PROC_TOKENS):
+        return {RichCompletion(prefix, append_space=True)}
+    return None
+
+
+@contextual_command_completer
+def complete_end_proc_keywords(command_context: CommandContext):
+    """If there's no space following an END_PROC_KEYWORD, insert one"""
+    if command_context.opening_quote or not command_context.prefix:
+        return None
+    prefix = command_context.prefix
+    if prefix in END_PROC_KEYWORDS:
+        return {RichCompletion(prefix, append_space=True)}
+    return None

xonsh/completers/completer.py

@@ -1,7 +1,8 @@
 import builtins
 import collections
+from xonsh.parsers.completion_context import CommandContext
 
-from xonsh.completers.tools import justify
+from xonsh.completers.tools import contextual_command_completer_for, justify
 import xonsh.lazyasd as xla
 
 
@@ -11,33 +12,31 @@ def xsh_session():
     return builtins.__xonsh__  # type: ignore
 
 
-def complete_completer(prefix, line, start, end, ctx):
+@contextual_command_completer_for("completer")
+def complete_completer(command: CommandContext):
     """
     Completion for "completer"
     """
-    args = line.split(" ")
-    if len(args) == 0 or args[0] != "completer":
-        return None
-
-    if end < len(line) and line[end] != " ":
+    if command.suffix:
         # completing in a middle of a word
         # (e.g. "completer some<TAB>thing")
         return None
 
-    curix = args.index(prefix)
+    curix = command.arg_index
 
     compnames = set(xsh_session.completers.keys())
     if curix == 1:
         possible = {"list", "help", "add", "remove"}
     elif curix == 2:
-        if args[1] == "help":
+        first_arg = command.args[1].value
+        if first_arg == "help":
             possible = {"list", "add", "remove"}
-        elif args[1] == "remove":
+        elif first_arg == "remove":
             possible = compnames
         else:
             raise StopIteration
     else:
-        if args[1] != "add":
+        if command.args[1].value != "add":
             raise StopIteration
         if curix == 3:
             possible = {i for i, j in xsh_session.ctx.items() if callable(j)}
@@ -49,7 +48,7 @@ def complete_completer(prefix, line, start, end, ctx):
             )
         else:
             raise StopIteration
-    return {i for i in possible if i.startswith(prefix)}
+    return {i for i in possible if i.startswith(command.prefix)}
 
 
 def add_one_completer(name, func, loc="end"):

xonsh/completers/dirs.py

@@ -1,31 +1,30 @@
 from xonsh.completers.man import complete_from_man
 from xonsh.completers.path import complete_dir
+from xonsh.completers.tools import contextual_command_completer_for
+from xonsh.parsers.completion_context import (
+    CompletionContext,
+    CommandContext,
+)
 
 
-def complete_cd(prefix, line, start, end, ctx):
+@contextual_command_completer_for("cd")
+def complete_cd(command: CommandContext):
     """
     Completion for "cd", includes only valid directory names.
     """
-    if start != 0 and line.split(" ")[0] == "cd":
-        results, prefix = complete_dir(prefix, line, start, end, ctx, True)
-        if len(results) == 0:
-            raise StopIteration
-        return results, prefix
-    return set()
+    results, lprefix = complete_dir(command)
+    if len(results) == 0:
+        raise StopIteration
+    return results, lprefix
 
 
-def complete_rmdir(prefix, line, start, end, ctx):
+@contextual_command_completer_for("rmdir")
+def complete_rmdir(command: CommandContext):
     """
     Completion for "rmdir", includes only valid directory names.
     """
-    if start != 0 and line.split(" ")[0] == "rmdir":
-        opts = {
-            i
-            for i in complete_from_man("-", "rmdir -", 6, 7, ctx)
-            if i.startswith(prefix)
-        }
-        comps, lp = complete_dir(prefix, line, start, end, ctx, True)
-        if len(comps) == 0 and len(opts) == 0:
-            raise StopIteration
-        return comps | opts, lp
-    return set()
+    opts = complete_from_man(CompletionContext(command))
+    comps, lp = complete_dir(command)
+    if len(comps) == 0 and len(opts) == 0:
+        raise StopIteration
+    return comps | opts, lp

xonsh/completers/init.py

@@ -10,9 +10,12 @@ from xonsh.completers.dirs import complete_cd, complete_rmdir
 from xonsh.completers.python import (
     complete_python,
     complete_import,
-    complete_python_mode,
 )
-from xonsh.completers.commands import complete_skipper
+from xonsh.completers.commands import (
+    complete_skipper,
+    complete_end_proc_tokens,
+    complete_end_proc_keywords,
+)
 from xonsh.completers.completer import complete_completer
 from xonsh.completers.xompletions import complete_xonfig, complete_xontrib
 
@@ -21,7 +24,7 @@ def default_completers():
     """Creates a copy of the default completers."""
     return collections.OrderedDict(
         [
-            ("python_mode", complete_python_mode),
+            ("end_proc_tokens", complete_end_proc_tokens),
             ("base", complete_base),
             ("completer", complete_completer),
             ("skip", complete_skipper),
@@ -35,5 +38,9 @@ def default_completers():
             ("import", complete_import),
             ("python", complete_python),
             ("path", complete_path),
+            (
+                "end_proc_keywords",
+                complete_end_proc_keywords,
+            ),  # this is last to give a chance to complete `and/or` prefixes
         ]
     )

xonsh/completers/man.py

@@ -5,9 +5,11 @@ import builtins
 import subprocess
 import typing as tp
 
+from xonsh.parsers.completion_context import CommandContext
+
 import xonsh.lazyasd as xl
 
-from xonsh.completers.tools import get_filter_function
+from xonsh.completers.tools import get_filter_function, contextual_command_completer
 
 OPTIONS: tp.Optional[tp.Dict[str, tp.Any]] = None
 OPTIONS_PATH: tp.Optional[str] = None
@@ -23,36 +25,37 @@ def INNER_OPTIONS_RE():
     return re.compile(r"-\w|--[a-z0-9-]+")
 
 
-def complete_from_man(prefix, line, start, end, ctx):
+@contextual_command_completer
+def complete_from_man(context: CommandContext):
     """
     Completes an option name, based on the contents of the associated man
     page.
     """
     global OPTIONS, OPTIONS_PATH
     if OPTIONS is None:
-        datadir = builtins.__xonsh__.env["XONSH_DATA_DIR"]
+        datadir: str = builtins.__xonsh__.env["XONSH_DATA_DIR"]  # type: ignore
         OPTIONS_PATH = os.path.join(datadir, "man_completions_cache")
         try:
             with open(OPTIONS_PATH, "rb") as f:
                 OPTIONS = pickle.load(f)
         except Exception:
             OPTIONS = {}
-    if not prefix.startswith("-"):
+    if context.arg_index == 0 or not context.prefix.startswith("-"):
         return set()
-    cmd = line.split()[0]
+    cmd = context.args[0].value
     if cmd not in OPTIONS:
         try:
             manpage = subprocess.Popen(
                 ["man", cmd], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL
             )
             # This is a trick to get rid of reverse line feeds
-            text = subprocess.check_output(["col", "-b"], stdin=manpage.stdout)
-            text = text.decode("utf-8")
+            enc_text = subprocess.check_output(["col", "-b"], stdin=manpage.stdout)
+            text = enc_text.decode("utf-8")
             scraped_text = " ".join(SCRAPE_RE.findall(text))
             matches = INNER_OPTIONS_RE.findall(scraped_text)
             OPTIONS[cmd] = matches
-            with open(OPTIONS_PATH, "wb") as f:
+            with open(tp.cast(str, OPTIONS_PATH), "wb") as f:
                 pickle.dump(OPTIONS, f)
         except Exception:
             return set()
-    return {s for s in OPTIONS[cmd] if get_filter_function()(s, prefix)}
+    return {s for s in OPTIONS[cmd] if get_filter_function()(s, context.prefix)}

xonsh/completers/path.py

@@ -3,12 +3,13 @@ import re
 import ast
 import glob
 import builtins
+from xonsh.parsers.completion_context import CommandContext
 
 import xonsh.tools as xt
 import xonsh.platform as xp
 import xonsh.lazyasd as xl
 
-from xonsh.completers.tools import get_filter_function
+from xonsh.completers.tools import RichCompletion, get_filter_function
 
 
 @xl.lazyobject
@@ -198,8 +199,6 @@ def _quote_paths(paths, start, end, append_end=True, cdpath=False):
         if end != "":
             if "r" not in start.lower():
                 s = s.replace(backslash, double_backslash)
-            if s.endswith(backslash) and not s.endswith(double_backslash):
-                s += backslash
         if end in s:
             s = s.replace(end, "".join("\\%s" % i for i in end))
         s = start + s + end if append_end else start + s
@@ -356,5 +355,27 @@ def complete_path(prefix, line, start, end, ctx, cdpath=True, filtfunc=None):
     return paths, lprefix
 
 
-def complete_dir(prefix, line, start, end, ctx, cdpath=False):
-    return complete_path(prefix, line, start, end, cdpath, filtfunc=os.path.isdir)
+def contextual_complete_path(command: CommandContext, cdpath=True, filtfunc=None):
+    # ``complete_path`` may add opening quotes:
+    prefix = command.raw_prefix
+
+    completions, lprefix = complete_path(
+        prefix,
+        prefix,
+        0,
+        len(prefix),
+        ctx={},
+        cdpath=cdpath,
+        filtfunc=filtfunc,
+    )
+
+    # ``complete_path`` may have added closing quotes:
+    rich_completions = {
+        RichCompletion(comp, append_closing_quote=False) for comp in completions
+    }
+
+    return rich_completions, lprefix
+
+
+def complete_dir(command: CommandContext):
+    return contextual_complete_path(command, filtfunc=os.path.isdir)

xonsh/completers/pip.py

@@ -5,16 +5,26 @@ import re
 import subprocess
 
 import xonsh.lazyasd as xl
+from xonsh.completers.tools import (
+    contextual_command_completer,
+    get_filter_function,
+    RichCompletion,
+)
+from xonsh.parsers.completion_context import CommandContext
+
+
+PIP_LIST_COMMANDS = {"uninstall", "show"}
 
 
 @xl.lazyobject
 def PIP_RE():
-    return re.compile(r"\bx?pip(?:\d|\.)*\b")
+    return re.compile(r"\bx?pip(?:\d|\.)*$")
 
 
 @xl.lazyobject
-def PIP_LIST_RE():
-    return re.compile(r"\bx?pip(?:\d|\.)*\b (?:uninstall|show)")
+def PACKAGE_IGNORE_PATTERN():
+    # These are the first and second lines in `pip list`'s output
+    return re.compile(r"^Package$|^-+$")
 
 
 @xl.lazyobject
@@ -24,33 +34,49 @@ def ALL_COMMANDS():
             subprocess.check_output(["pip", "--help"], stderr=subprocess.DEVNULL)
         )
     except FileNotFoundError:
-        return []
+        try:
+            help_text = str(
+                subprocess.check_output(["pip3", "--help"], stderr=subprocess.DEVNULL)
+            )
+        except FileNotFoundError:
+            return []
     commands = re.findall(r"  (\w+)  ", help_text)
     return [c for c in commands if c not in ["completion", "help"]]
 
 
-def complete_pip(prefix, line, begidx, endidx, ctx):
+@contextual_command_completer
+def complete_pip(context: CommandContext):
     """Completes python's package manager pip"""
-    line_len = len(line.split())
-    if (
-        (line_len > 3)
-        or (line_len > 2 and line.endswith(" "))
-        or (not PIP_RE.search(line))
-    ):
-        return
-    if PIP_LIST_RE.search(line):
-        try:
-            items = subprocess.check_output(["pip", "list"], stderr=subprocess.DEVNULL)
-        except FileNotFoundError:
-            return set()
-        items = items.decode("utf-8").splitlines()
-        return set(i.split()[0] for i in items if i.split()[0].startswith(prefix))
+    prefix = context.prefix
+    if context.arg_index == 0 or (not PIP_RE.search(context.args[0].value)):
+        return None
+    filter_func = get_filter_function()
 
-    if (line_len > 1 and line.endswith(" ")) or line_len > 2:
-        # "pip show " -> no complete (note space)
-        return
-    if prefix not in ALL_COMMANDS:
-        suggestions = [c for c in ALL_COMMANDS if c.startswith(prefix)]
-        if suggestions:
-            return suggestions, len(prefix)
-    return ALL_COMMANDS, len(prefix)
+    if context.arg_index == 2 and context.args[1].value in PIP_LIST_COMMANDS:
+        # `pip show PREFIX` - complete package names
+        try:
+            enc_items = subprocess.check_output(
+                [context.args[0].value, "list"], stderr=subprocess.DEVNULL
+            )
+        except FileNotFoundError:
+            return None
+        packages = (
+            line.split(maxsplit=1)[0] for line in enc_items.decode("utf-8").splitlines()
+        )
+        return {
+            package
+            for package in packages
+            if filter_func(package, prefix)
+            and not PACKAGE_IGNORE_PATTERN.match(package)
+        }
+
+    if context.arg_index == 1:
+        # `pip PREFIX` - complete pip commands
+        suggestions = {
+            RichCompletion(c, append_space=True)
+            for c in ALL_COMMANDS
+            if filter_func(c, prefix)
+        }
+        return suggestions
+
+    return None

xonsh/completers/python.py

@@ -6,11 +6,17 @@ import builtins
 import importlib
 import warnings
 import collections.abc as cabc
+from xonsh.parsers.completion_context import CompletionContext, PythonContext
 
 import xonsh.tools as xt
 import xonsh.lazyasd as xl
 
-from xonsh.completers.tools import get_filter_function
+from xonsh.completers.tools import (
+    CompleterResult,
+    contextual_completer,
+    get_filter_function,
+    RichCompletion,
+)
 
 
 @xl.lazyobject
@@ -21,15 +27,15 @@ def RE_ATTR():
 @xl.lazyobject
 def XONSH_EXPR_TOKENS():
     return {
-        "and ",
+        RichCompletion("and", append_space=True),
         "else",
-        "for ",
-        "if ",
-        "in ",
-        "is ",
-        "lambda ",
-        "not ",
-        "or ",
+        RichCompletion("for", append_space=True),
+        RichCompletion("if", append_space=True),
+        RichCompletion("in", append_space=True),
+        RichCompletion("is", append_space=True),
+        RichCompletion("lambda", append_space=True),
+        RichCompletion("not", append_space=True),
+        RichCompletion("or", append_space=True),
         "+",
         "-",
         "/",
@@ -48,7 +54,7 @@ def XONSH_EXPR_TOKENS():
         ">=",
         "==",
         "!=",
-        ",",
+        RichCompletion(",", append_space=True),
         "?",
         "??",
         "$(",
@@ -66,27 +72,27 @@ def XONSH_EXPR_TOKENS():
 @xl.lazyobject
 def XONSH_STMT_TOKENS():
     return {
-        "as ",
-        "assert ",
+        RichCompletion("as", append_space=True),
+        RichCompletion("assert", append_space=True),
         "break",
-        "class ",
+        RichCompletion("class", append_space=True),
         "continue",
-        "def ",
-        "del ",
-        "elif ",
-        "except ",
+        RichCompletion("def", append_space=True),
+        RichCompletion("del", append_space=True),
+        RichCompletion("elif", append_space=True),
+        RichCompletion("except", append_space=True),
         "finally:",
-        "from ",
-        "global ",
-        "import ",
-        "nonlocal ",
+        RichCompletion("from", append_space=True),
+        RichCompletion("global", append_space=True),
+        RichCompletion("import", append_space=True),
+        RichCompletion("nonlocal", append_space=True),
         "pass",
-        "raise ",
-        "return ",
+        RichCompletion("raise", append_space=True),
+        RichCompletion("return", append_space=True),
         "try:",
-        "while ",
-        "with ",
-        "yield ",
+        RichCompletion("while", append_space=True),
+        RichCompletion("with", append_space=True),
+        RichCompletion("yield", append_space=True),
         "-",
         "/",
         "//",
@@ -125,34 +131,45 @@ def XONSH_TOKENS():
     return set(XONSH_EXPR_TOKENS) | set(XONSH_STMT_TOKENS)
 
 
-def complete_python(prefix, line, start, end, ctx):
+@contextual_completer
+def complete_python(context: CompletionContext) -> CompleterResult:
     """
     Completes based on the contents of the current Python environment,
     the Python built-ins, and xonsh operators.
     If there are no matches, split on common delimiters and try again.
     """
-    rtn = _complete_python(prefix, line, start, end, ctx)
+    if context.python is None:
+        return None
+
+    if context.command and context.command.arg_index != 0:
+        # this can be a command (i.e. not a subexpression)
+        first = context.command.args[0].value
+        ctx = context.python.ctx or {}
+        if first in builtins.__xonsh__.commands_cache and first not in ctx:  # type: ignore
+            # this is a known command, so it won't be python code
+            return None
+
+    line = context.python.multiline_code
+    prefix = (line.rsplit(maxsplit=1) or [""])[-1]
+    rtn = _complete_python(prefix, context.python)
     if not rtn:
         prefix = (
             re.split(r"\(|=|{|\[|,", prefix)[-1]
             if not prefix.startswith(",")
             else prefix
         )
-        start = line.find(prefix)
-        rtn = _complete_python(prefix, line, start, end, ctx)
-        return rtn, len(prefix)
-    return rtn
+        rtn = _complete_python(prefix, context.python)
+    return rtn, len(prefix)
 
 
-def _complete_python(prefix, line, start, end, ctx):
+def _complete_python(prefix, context: PythonContext):
     """
     Completes based on the contents of the current Python environment,
     the Python built-ins, and xonsh operators.
     """
-    if line != "":
-        first = line.split()[0]
-        if first in builtins.__xonsh__.commands_cache and first not in ctx:
-            return set()
+    line = context.multiline_code
+    end = context.cursor_index
+    ctx = context.ctx
     filt = get_filter_function()
     rtn = set()
     if ctx is not None:
@@ -172,19 +189,6 @@ def _complete_python(prefix, line, start, end, ctx):
     return rtn
 
 
-def complete_python_mode(prefix, line, start, end, ctx):
-    """
-    Python-mode completions for @( and ${
-    """
-    if not (prefix.startswith("@(") or prefix.startswith("${")):
-        return set()
-    prefix_start = prefix[:2]
-    python_matches = complete_python(prefix[2:], line, start - 2, end - 2, ctx)
-    if isinstance(python_matches, cabc.Sequence):
-        python_matches = python_matches[0]
-    return set(prefix_start + i for i in python_matches)
-
-
 def _turn_off_warning(func):
     """Decorator to turn off warning temporarily."""
 
@@ -277,23 +281,37 @@ def python_signature_complete(prefix, line, end, ctx, filter_func):
     return args
 
 
-def complete_import(prefix, line, start, end, ctx):
+@contextual_completer
+def complete_import(context: CompletionContext):
     """
     Completes module names and contents for "import ..." and "from ... import
     ..."
     """
-    ltoks = line.split()
-    ntoks = len(ltoks)
-    if ntoks == 2 and ltoks[0] == "from":
+    if not (context.command and context.python):
+        # Imports are only possible in independent lines (not in `$()` or `@()`).
+        # This means it's python code, but also can be a command as far as the parser is concerned.
+        return None
+
+    command = context.command
+
+    if command.opening_quote:
+        # can't have a quoted import
+        return None
+
+    arg_index = command.arg_index
+    prefix = command.prefix
+    args = command.args
+
+    if arg_index == 1 and args[0].value == "from":
         # completing module to import
         return {"{} ".format(i) for i in complete_module(prefix)}
-    if ntoks > 1 and ltoks[0] == "import" and start == len("import "):
+    if arg_index >= 1 and args[0].value == "import":
         # completing module to import
         return complete_module(prefix)
-    if ntoks > 2 and ltoks[0] == "from" and ltoks[2] == "import":
+    if arg_index > 2 and args[0].value == "from" and args[2].value == "import":
         # complete thing inside a module
         try:
-            mod = importlib.import_module(ltoks[1])
+            mod = importlib.import_module(args[1].value)
         except ImportError:
             return set()
         out = {i[0] for i in inspect.getmembers(mod) if i[0].startswith(prefix)}

xonsh/completers/tools.py

@@ -1,6 +1,10 @@
 """Xonsh completer tools."""
 import builtins
 import textwrap
+import typing as tp
+from functools import wraps
+
+from xonsh.parsers.completion_context import CompletionContext, CommandContext
 
 
 def _filter_normal(s, x):
@@ -34,51 +38,121 @@ def justify(s, max_length, left_pad=0):
 
 
 class RichCompletion(str):
-    """A rich completion that completers can return instead of a string
+    """A rich completion that completers can return instead of a string"""
 
-    Parameters
-    ----------
-    value : str
-        The completion's actual value.
-    prefix_len : int
-        Length of the prefix to be replaced in the completion.
-        If None, the default prefix len will be used.
-    display : str
-        Text to display in completion option list.
-        If None, ``value`` will be used.
-    description : str
-        Extra text to display when the completion is selected.
-    """
-
-    def __new__(cls, value, prefix_len=None, display=None, description="", style=""):
+    def __new__(cls, value, *args, **kwargs):
         completion = super().__new__(cls, value)
-
-        completion.prefix_len = prefix_len
-        completion.display = display or value
-        completion.description = description
-        completion.style = style
-
+        # ``str``'s ``__new__`` doesn't call ``__init__``, so we'll call it ourselves
+        cls.__init__(completion, value, *args, **kwargs)
         return completion
 
+    def __init__(
+        self,
+        value: str,
+        prefix_len: tp.Optional[int] = None,
+        display: tp.Optional[str] = None,
+        description: str = "",
+        style: str = "",
+        append_closing_quote: bool = True,
+        append_space: bool = False,
+    ):
+        """
+        Parameters
+        ----------
+        value :
+            The completion's actual value.
+        prefix_len :
+            Length of the prefix to be replaced in the completion.
+            If None, the default prefix len will be used.
+        display :
+            Text to display in completion option list instead of ``value``.
+            NOTE: If supplied, the common prefix with other completions won't be removed.
+        description :
+            Extra text to display when the completion is selected.
+        style :
+            Style to pass to prompt-toolkit's ``Completion`` object.
+        append_closing_quote :
+            Whether to append a closing quote to the completion if the cursor is after it.
+            See ``Completer.complete`` in ``xonsh/completer.py``
+        append_space :
+            Whether to append a space after the completion.
+            This is intended to work with ``appending_closing_quote``, so the space will be added correctly **after** the closing quote.
+            This is used in ``Completer.complete``.
+            An extra bonus is that the space won't show up in the ``display`` attribute.
+        """
+        super().__init__()
+        self.prefix_len = prefix_len
+        self.display = display
+        self.description = description
+        self.style = style
+        self.append_closing_quote = append_closing_quote
+        self.append_space = append_space
+
+    @property
+    def value(self):
+        return str(self)
+
     def __repr__(self):
-        return "RichCompletion({}, prefix_len={}, display={}, description={})".format(
-            repr(str(self)), self.prefix_len, repr(self.display), repr(self.description)
+        attrs = ", ".join(f"{attr}={value!r}" for attr, value in self.__dict__.items())
+        return f"RichCompletion({self.value!r}, {attrs})"
+
+    def replace(self, **kwargs):
+        """Create a new RichCompletion with replaced attributes"""
+        default_kwargs = dict(
+            value=self.value,
+            **self.__dict__,
         )
+        default_kwargs.update(kwargs)
+        return RichCompletion(**default_kwargs)
 
 
-def get_ptk_completer():
-    """Get the current PromptToolkitCompleter
+Completion = tp.Union[RichCompletion, str]
+CompleterResult = tp.Union[tp.Set[Completion], tp.Tuple[tp.Set[Completion], int], None]
+ContextualCompleter = tp.Callable[[CompletionContext], CompleterResult]
 
-    This is usefull for completers that want to use
-    PromptToolkitCompleter.current_document (the current multiline document).
 
-    Call this function lazily since in '.xonshrc' the shell doesn't exist.
+def contextual_completer(func: ContextualCompleter):
+    """Decorator for a contextual completer
 
-    Returns
-    -------
-    The PromptToolkitCompleter if running with ptk, else returns None
+    This is used to mark completers that want to use the parsed completion context.
+    See ``xonsh/parsers/completion_context.py``.
+
+    ``func`` receives a single CompletionContext object.
     """
-    if __xonsh__.shell is None or __xonsh__.shell.shell_type != "prompt_toolkit":
+    func.contextual = True  # type: ignore
+    return func
+
+
+def is_contextual_completer(func):
+    return getattr(func, "contextual", False)
+
+
+def contextual_command_completer(func: tp.Callable[[CommandContext], CompleterResult]):
+    """like ``contextual_completer``,
+    but will only run when completing a command and will directly receive the ``CommandContext`` object"""
+
+    @contextual_completer
+    @wraps(func)
+    def _completer(context: CompletionContext) -> CompleterResult:
+        if context.command is not None:
+            return func(context.command)
         return None
 
-    return __xonsh__.shell.shell.pt_completer
+    return _completer
+
+
+def contextual_command_completer_for(cmd: str):
+    """like ``contextual_command_completer``,
+    but will only run when completing the ``cmd`` command"""
+
+    def decor(func: tp.Callable[[CommandContext], CompleterResult]):
+        @contextual_completer
+        @wraps(func)
+        def _completer(context: CompletionContext) -> CompleterResult:
+            if context.command is not None and context.command.completing_command(cmd):
+                return func(context.command)
+            return None
+
+        return _completer
+
+    return decor

xonsh/completers/xompletions.py

@@ -1,24 +1,24 @@
 """Provides completions for xonsh internal utilities"""
 
+from xonsh.parsers.completion_context import CommandContext
 import xonsh.xontribs as xx
 import xonsh.xontribs_meta as xmt
 import xonsh.tools as xt
 from xonsh.xonfig import XONFIG_MAIN_ACTIONS
+from xonsh.completers.tools import contextual_command_completer_for
 
 
-def complete_xonfig(prefix, line, start, end, ctx):
+@contextual_command_completer_for("xonfig")
+def complete_xonfig(command: CommandContext):
     """Completion for ``xonfig``"""
-    args = line.split(" ")
-    if len(args) == 0 or args[0] != "xonfig":
-        return None
-    curix = args.index(prefix)
+    curix = command.arg_index
     if curix == 1:
         possible = set(XONFIG_MAIN_ACTIONS.keys()) | {"-h"}
-    elif curix == 2 and args[1] == "colors":
+    elif curix == 2 and command.args[1].value == "colors":
         possible = set(xt.color_style_names())
     else:
         raise StopIteration
-    return {i for i in possible if i.startswith(prefix)}
+    return {i for i in possible if i.startswith(command.prefix)}
 
 
 def _list_installed_xontribs():
@@ -32,18 +32,15 @@ def _list_installed_xontribs():
     return installed
 
 
-def complete_xontrib(prefix, line, start, end, ctx):
+@contextual_command_completer_for("xontrib")
+def complete_xontrib(command: CommandContext):
     """Completion for ``xontrib``"""
-    args = line.split(" ")
-    if len(args) == 0 or args[0] != "xontrib":
-        return None
-    curix = args.index(prefix)
+    curix = command.arg_index
     if curix == 1:
         possible = {"list", "load"}
-    elif curix == 2:
-        if args[1] == "load":
-            possible = _list_installed_xontribs()
+    elif curix == 2 and command.args[1].value == "load":
+        possible = _list_installed_xontribs()
     else:
         raise StopIteration
 
-    return {i for i in possible if i.startswith(prefix)}
+    return {i for i in possible if i.startswith(command.prefix)}

xonsh/jupyter_kernel.py

@@ -418,15 +418,40 @@ class XonshKernel:
             identities=identities,
         )
 
-    def do_complete(self, code, pos):
+    def do_complete(self, code: str, pos: int):
         """Get completions."""
-        shell = builtins.__xonsh__.shell
-        line = code.split("\n")[-1]
-        line = builtins.aliases.expand_alias(line)
-        prefix = line.split(" ")[-1]
-        endidx = pos
-        begidx = pos - len(prefix)
-        rtn, _ = self.completer.complete(prefix, line, begidx, endidx, shell.ctx)
+        shell = builtins.__xonsh__.shell  # type: ignore
+        line_start = code.rfind("\n", 0, pos) + 1
+        line_stop = code.find("\n", pos)
+        if line_stop == -1:
+            line_stop = len(code)
+        else:
+            line_stop += 1
+        line = code[line_start:line_stop]
+        endidx = pos - line_start
+        line_ex: str = builtins.aliases.expand_alias(line, endidx)  # type: ignore
+
+        begidx = line[:endidx].rfind(" ") + 1 if line[:endidx].rfind(" ") >= 0 else 0
+        prefix = line[begidx:endidx]
+        expand_offset = len(line_ex) - len(line)
+
+        multiline_text = code
+        cursor_index = pos
+        if line != line_ex:
+            multiline_text = (
+                multiline_text[:line_start] + line_ex + multiline_text[line_stop:]
+            )
+            cursor_index += expand_offset
+
+        rtn, _ = self.completer.complete(
+            prefix,
+            line_ex,
+            begidx + expand_offset,
+            endidx + expand_offset,
+            shell.ctx,
+            multiline_text=multiline_text,
+            cursor_index=cursor_index,
+        )
         if isinstance(rtn, Set):
             rtn = list(rtn)
         message = {

xonsh/lexer.py

@@ -168,7 +168,7 @@ def handle_rparen(state, token):
     Function for handling ``)``
     """
     e = _end_delimiter(state, token)
-    if e is None:
+    if e is None or state["tolerant"]:
         state["last"] = token
         yield _new_token("RPAREN", ")", token.start)
     else:
@@ -178,7 +178,7 @@ def handle_rparen(state, token):
 def handle_rbrace(state, token):
     """Function for handling ``}``"""
     e = _end_delimiter(state, token)
-    if e is None:
+    if e is None or state["tolerant"]:
         state["last"] = token
         yield _new_token("RBRACE", "}", token.start)
     else:
@@ -190,7 +190,7 @@ def handle_rbracket(state, token):
     Function for handling ``]``
     """
     e = _end_delimiter(state, token)
-    if e is None:
+    if e is None or state["tolerant"]:
         state["last"] = token
         yield _new_token("RBRACKET", "]", token.start)
     else:
@@ -365,7 +365,7 @@ def handle_token(state, token):
         yield _new_token("ERRORTOKEN", m, token.start)
 
 
-def get_tokens(s):
+def get_tokens(s, tolerant):
     """
     Given a string containing xonsh code, generates a stream of relevant PLY
     tokens using ``handle_token``.
@@ -374,14 +374,15 @@ def get_tokens(s):
         "indents": [0],
         "last": None,
         "pymode": [(True, "", "", (0, 0))],
-        "stream": tokenize(io.BytesIO(s.encode("utf-8")).readline),
+        "stream": tokenize(io.BytesIO(s.encode("utf-8")).readline, tolerant),
+        "tolerant": tolerant,
     }
     while True:
         try:
             token = next(state["stream"])
             yield from handle_token(state, token)
         except StopIteration:
-            if len(state["pymode"]) > 1:
+            if len(state["pymode"]) > 1 and not tolerant:
                 pm, o, m, p = state["pymode"][-1]
                 l, c = p
                 e = 'Unmatched "{}" at line {}, column {}'
@@ -412,7 +413,7 @@ class Lexer(object):
 
     _tokens: tp.Optional[tp.Tuple[str, ...]] = None
 
-    def __init__(self):
+    def __init__(self, tolerant=False):
         """
         Attributes
         ----------
@@ -422,11 +423,15 @@ class Lexer(object):
             The last token seen.
         lineno : int
             The last line number seen.
+        tolerant : bool
+            Tokenize without extra checks (e.g. paren matching).
+            When True, ERRORTOKEN contains the erroneous string instead of an error msg.
 
         """
         self.fname = ""
         self.last = None
         self.beforelast = None
+        self.tolerant = tolerant
 
     def build(self, **kwargs):
         """Part of the PLY lexer API."""
@@ -437,7 +442,7 @@ class Lexer(object):
 
     def input(self, s):
         """Calls the lexer on the string s."""
-        self.token_stream = get_tokens(s)
+        self.token_stream = get_tokens(s, self.tolerant)
 
     def token(self):
         """Retrieves the next token."""

xonsh/parsers/base.py

@@ -215,6 +215,29 @@ def hasglobstar(x):
         return False
 
 
+def raise_parse_error(
+    msg: tp.Union[str, tp.Tuple[str]],
+    loc: tp.Optional[Location] = None,
+    code: tp.Optional[str] = None,
+    lines: tp.Optional[tp.List[str]] = None,
+):
+    if loc is None or code is None or lines is None:
+        err_line_pointer = ""
+    else:
+        col = loc.column + 1
+        if loc.lineno == 0:
+            loc.lineno = len(lines)
+        i = loc.lineno - 1
+        if 0 <= i < len(lines):
+            err_line = lines[i].rstrip()
+            err_line_pointer = "\n{}\n{: >{}}".format(err_line, "^", col)
+        else:
+            err_line_pointer = ""
+    err = SyntaxError("{0}: {1}{2}".format(loc, msg, err_line_pointer))
+    err.loc = loc  # type: ignore
+    raise err
+
+
 class YaccLoader(Thread):
     """Thread to load (but not shave) the yacc parser."""
 
@@ -630,22 +653,7 @@ class BaseParser(object):
         raise SyntaxError()
 
     def _parse_error(self, msg, loc):
-        if self.xonsh_code is None or loc is None:
-            err_line_pointer = ""
-        else:
-            col = loc.column + 1
-            lines = self.lines
-            if loc.lineno == 0:
-                loc.lineno = len(lines)
-            i = loc.lineno - 1
-            if 0 <= i < len(lines):
-                err_line = lines[i].rstrip()
-                err_line_pointer = "\n{}\n{: >{}}".format(err_line, "^", col)
-            else:
-                err_line_pointer = ""
-        err = SyntaxError("{0}: {1}{2}".format(loc, msg, err_line_pointer))
-        err.loc = loc
-        raise err
+        raise_parse_error(msg, loc, self.xonsh_code, self.lines)
 
     #
     # Precedence of operators

xonsh/ptk_shell/completer.py

@@ -24,7 +24,6 @@ class PromptToolkitCompleter(Completer):
         self.ctx = ctx
         self.shell = shell
         self.hist_suggester = AutoSuggestFromHistory()
-        self.current_document = None
 
     def get_completions(self, document, complete_event):
         """Returns a generator for list of completions."""
@@ -36,24 +35,37 @@ class PromptToolkitCompleter(Completer):
         if not should_complete or self.completer is None:
             return
         # generate actual completions
-        line = document.current_line.lstrip()
-        line_ex = builtins.aliases.expand_alias(line)
+        line = document.current_line
 
         endidx = document.cursor_position_col
+        line_ex = builtins.aliases.expand_alias(line, endidx)
+
         begidx = line[:endidx].rfind(" ") + 1 if line[:endidx].rfind(" ") >= 0 else 0
         prefix = line[begidx:endidx]
         expand_offset = len(line_ex) - len(line)
 
-        # enable completers to access entire document
-        self.current_document = document
+        multiline_text = document.text
+        cursor_index = document.cursor_position
+        if line != line_ex:
+            line_start = cursor_index - len(document.current_line_before_cursor)
+            multiline_text = (
+                multiline_text[:line_start]
+                + line_ex
+                + multiline_text[line_start + len(line) :]
+            )
+            cursor_index += expand_offset
 
         # get normal completions
         completions, plen = self.completer.complete(
-            prefix, line_ex, begidx + expand_offset, endidx + expand_offset, self.ctx
+            prefix,
+            line_ex,
+            begidx + expand_offset,
+            endidx + expand_offset,
+            self.ctx,
+            multiline_text=multiline_text,
+            cursor_index=cursor_index,
         )
 
-        self.current_document = None
-
         # completions from auto suggest
         sug_comp = None
         if env.get("AUTO_SUGGEST") and env.get("AUTO_SUGGEST_IN_COMPLETIONS"):
@@ -89,7 +101,7 @@ class PromptToolkitCompleter(Completer):
                 yield Completion(
                     comp,
                     -comp.prefix_len if comp.prefix_len is not None else -plen,
-                    display=comp.display,
+                    display=comp.display or comp[pre:].strip("'\""),
                     display_meta=comp.description or None,
                     style=comp.style or "",
                 )

xonsh/readline_shell.py

@@ -426,8 +426,15 @@ class ReadlineShell(BaseShell, cmd.Cmd):
         rl_completion_suppress_append()  # this needs to be called each time
         _rebind_case_sensitive_completions()
         rl_completion_query_items(val=999999999)
+        prev_text = "".join(self.buffer)
         completions, plen = self.completer.complete(
-            prefix, line, begidx, endidx, ctx=self.ctx
+            prefix,
+            line,
+            begidx,
+            endidx,
+            ctx=self.ctx,
+            multiline_text=prev_text + line,
+            cursor_index=len(prev_text) + endidx,
         )
         rtn_completions = _render_completions(completions, prefix, plen)
 

xonsh/tokenize.py

@@ -857,7 +857,7 @@ def tokopen(filename):
         raise
 
 
-def _tokenize(readline, encoding):
+def _tokenize(readline, encoding, tolerant=False):
     lnum = parenlev = continued = 0
     numchars = "0123456789"
     contstr, needcont = "", 0
@@ -888,7 +888,14 @@ def _tokenize(readline, encoding):
 
         if contstr:  # continued string
             if not line:
-                raise TokenError("EOF in multi-line string", strstart)
+                if tolerant:
+                    # return the partial string
+                    yield TokenInfo(
+                        ERRORTOKEN, contstr, strstart, (lnum, end), contline + line
+                    )
+                    break
+                else:
+                    raise TokenError("EOF in multi-line string", strstart)
             endmatch = endprog.match(line)
             if endmatch:
                 pos = end = endmatch.end(0)
@@ -954,7 +961,9 @@ def _tokenize(readline, encoding):
                 indents.append(column)
                 yield TokenInfo(INDENT, line[:pos], (lnum, 0), (lnum, pos), line)
             while column < indents[-1]:
-                if column not in indents:
+                if (
+                    column not in indents and not tolerant
+                ):  # if tolerant, just ignore the error
                     raise IndentationError(
                         "unindent does not match any outer indentation level",
                         ("<tokenize>", lnum, pos, line),
@@ -975,6 +984,9 @@ def _tokenize(readline, encoding):
 
         else:  # continued statement
             if not line:
+                if tolerant:
+                    # no need to raise an error, we're done
+                    break
                 raise TokenError("EOF in multi-line statement", (lnum, 0))
             continued = 0
 
@@ -1117,7 +1129,7 @@ def _tokenize(readline, encoding):
     yield TokenInfo(ENDMARKER, "", (lnum, 0), (lnum, 0), "")
 
 
-def tokenize(readline):
+def tokenize(readline, tolerant=False):
     """
     The tokenize() generator requires one argument, readline, which
     must be a callable object which provides the same interface as the
@@ -1135,11 +1147,16 @@ def tokenize(readline):
 
     The first token sequence will always be an ENCODING token
     which tells you which encoding was used to decode the bytes stream.
+
+    If ``tolerant`` is True, yield ERRORTOKEN with the erroneous string instead of
+    throwing an exception when encountering an error.
     """
     encoding, consumed = detect_encoding(readline)
     rl_gen = iter(readline, b"")
     empty = itertools.repeat(b"")
-    return _tokenize(itertools.chain(consumed, rl_gen, empty).__next__, encoding)
+    return _tokenize(
+        itertools.chain(consumed, rl_gen, empty).__next__, encoding, tolerant
+    )
 
 
 # An undocumented, backwards compatible, API for all the places in the standard

xontrib/jedi.py

@@ -1,14 +1,16 @@
 """Use Jedi as xonsh's python completer."""
-import itertools
+import builtins
 
 import xonsh
+from xonsh.built_ins import XonshSession
 from xonsh.lazyasd import lazyobject, lazybool
 from xonsh.completers.tools import (
     get_filter_function,
-    get_ptk_completer,
     RichCompletion,
+    contextual_completer,
 )
 from xonsh.completers import _aliases
+from xonsh.parsers.completion_context import CompletionContext
 
 __all__ = ()
 
@@ -17,11 +19,6 @@ __all__ = ()
 import jedi
 
 
-@lazyobject
-def PTK_COMPLETER():
-    return get_ptk_completer()
-
-
 @lazybool
 def JEDI_NEW_API():
     if hasattr(jedi, "__version__"):
@@ -46,32 +43,40 @@ def XONSH_SPECIAL_TOKENS():
     }
 
 
-def complete_jedi(prefix, line, start, end, ctx):
+@lazyobject
+def XONSH_SPECIAL_TOKENS_FIRST():
+    return {tok[0] for tok in XONSH_SPECIAL_TOKENS}
+
+
+@contextual_completer
+def complete_jedi(context: CompletionContext):
     """Completes python code using Jedi and xonsh operators"""
+    if context.python is None:
+        return None
+
+    xonsh_execer: XonshSession = builtins.__xonsh__  # type: ignore
+    ctx = context.python.ctx or {}
 
     # if this is the first word and it's a known command, don't complete.
     # taken from xonsh/completers/python.py
-    if line.lstrip() != "":
-        first = line.split(maxsplit=1)[0]
-        if prefix == first and first in __xonsh__.commands_cache and first not in ctx:
-            return set()
+    if context.command and context.command.arg_index != 0:
+        first = context.command.args[0].value
+        if first in xonsh_execer.commands_cache and first not in ctx:  # type: ignore
+            return None
 
     filter_func = get_filter_function()
-    jedi.settings.case_insensitive_completion = not __xonsh__.env.get(
+    jedi.settings.case_insensitive_completion = not xonsh_execer.env.get(
         "CASE_SENSITIVE_COMPLETIONS"
     )
 
-    if PTK_COMPLETER:  # 'is not None' won't work with lazyobject
-        document = PTK_COMPLETER.current_document
-        source = document.text
-        row = document.cursor_position_row + 1
-        column = document.cursor_position_col
-    else:
-        source = line
-        row = 1
-        column = end
+    source = context.python.multiline_code
+    index = context.python.cursor_index
+    row = source.count("\n", 0, index) + 1
+    column = (
+        index - source.rfind("\n", 0, index) - 1
+    )  # will be `index - (-1) - 1` if there's no newline
 
-    extra_ctx = {"__xonsh__": __xonsh__}
+    extra_ctx = {"__xonsh__": xonsh_execer}
     try:
         extra_ctx["_"] = _
     except NameError:
@@ -91,22 +96,32 @@ def complete_jedi(prefix, line, start, end, ctx):
     except Exception:
         pass
 
-    # make sure _* names are completed only when
-    # the user writes the first underscore
-    complete_underscores = prefix.endswith("_")
+    res = set(create_completion(comp) for comp in script_comp if should_complete(comp))
 
-    return set(
-        itertools.chain(
-            (
-                create_completion(comp)
-                for comp in script_comp
-                if complete_underscores
-                or not comp.name.startswith("_")
-                or not comp.complete.startswith("_")
-            ),
-            (t for t in XONSH_SPECIAL_TOKENS if filter_func(t, prefix)),
+    if index > 0:
+        last_char = source[index - 1]
+        res.update(
+            RichCompletion(t, prefix_len=1)
+            for t in XONSH_SPECIAL_TOKENS
+            if filter_func(t, last_char)
         )
-    )
+    else:
+        res.update(RichCompletion(t, prefix_len=0) for t in XONSH_SPECIAL_TOKENS)
+
+    return res
+
+
+def should_complete(comp: jedi.api.classes.Completion):
+    """
+    make sure _* names are completed only when
+    the user writes the first underscore
+    """
+    name = comp.name
+    if not name.startswith("_"):
+        return True
+    completion = comp.complete
+    # only if we're not completing the first underscore:
+    return completion and len(completion) <= len(name) - 1
 
 
 def create_completion(comp: jedi.api.classes.Completion):
@@ -138,6 +153,5 @@ def create_completion(comp: jedi.api.classes.Completion):
 xonsh.completers.base.complete_python = complete_jedi
 
 # Jedi ignores leading '@(' and friends
-_aliases._remove_completer(["python_mode"])
 _aliases._add_one_completer("jedi_python", complete_jedi, "<python")
 _aliases._remove_completer(["python"])