Merge branch 'master' into rtnjdei

.appveyor.yml

@@ -1,4 +1,4 @@
-version: 0.5.3.{build}
+version: 0.5.5.{build}
 os: Windows Server 2012 R2
 environment:
 

.circle.yml

@@ -24,3 +24,5 @@ test:
     override:
         - case $CIRCLE_NODE_INDEX in 0) py.test --flake8 --timeout=10 --cov=./xonsh;; 1) py.test --timeout=10 ;; esac:
             parallel: true
+    post:
+        - if [[ $CIRCLE_NODE_INDEX -eq 0  ]]; then codecov; fi

.gitignore

@@ -13,6 +13,7 @@ tests/lexer_table.py
 tests/parser_table.py
 tests/lexer_test_table.py
 tests/parser_test_table.py
+tests/testfile
 build/
 dist/
 xonsh.egg-info/

.travis.yml

@@ -5,12 +5,14 @@ env:
 matrix:
     include:
         - os: linux
-          python: 3.5 
-          env: 
+          python: 3.5
+          env:
             - MINICONDA_OS="Linux"
+            - CI=true
+            - TRAVIS=true
         - os: linux
           python: 3.6
-          env: 
+          env:
             - MINICONDA_OS="Linux"
             - BUILD_DOCS=true
         - os: linux

CHANGELOG.rst

@@ -4,6 +4,110 @@ Xonsh Change Log
 
 .. current developments
 
+v0.5.5
+====================
+
+**Added:**
+
+* New ``--rc`` command line option allows users to specify paths to run control
+  files from the command line. This includes both xonsh-based and JSON-based
+  configuration.
+* New ``$UPDATE_COMPLETIONS_ON_KEYPRESS`` controls whether or not completions
+  will automatically display and update while typing. This feature is only
+  available in the prompt-toolkit shell.
+
+
+**Changed:**
+
+* Xonsh scripts now report ``__file__`` and ``__name__`` when run as scripts
+  or sourced. These variables have the same meaning as they do in Python
+  scripts.
+* ``$XONSHRC`` and related configuration variables now accept JSON-based
+  static configuration file names as elements. This unifies the two methods
+  of run control to a single entry point and loading system.
+* The ``xonsh.shell.Shell()`` class now requires that an Execer instance
+  be explicitly provided to its init method. This class is no longer
+  responsible for creating an execer an its deprendencies.
+* Moved decorators ``unthreadable``, ``uncapturable`` from
+  ``xonsh.proc`` to ``xonsh.tools``.
+* Some refactorings on jobs control.
+
+
+**Deprecated:**
+
+* The ``--config-path`` command line option is now deprecated in favor of
+  ``--rc``.
+
+
+**Removed:**
+
+* ``xonsh.environ.DEFAULT_XONSHRC`` has been removed due to deprecation.
+  For this value, please check the environment instead, or call
+  ``xonsh.environ.default_xonshrc(env)``.
+
+
+**Fixed:**
+
+* Command pipelines that end in a callable alias are now interruptable with
+  ``^C`` and the processes that are piped into the alais have their file handles
+  closed. This should ensure that the entire pipeline is closed.
+* Fixed issue where unthreadable subprocs were not allowed to be
+  captured with the ``$(cmd)`` operator.
+* The ``ProcProxy`` class (unthreadable aliases) was not being executed and would
+  hange if the alias was capturable. This has been fixed.
+* Fixed a ``tcsetattr: Interrupted system call`` issue when run xonsh scripts.
+* Fixed issue with ``ValueError`` being thrown from ``inspect.signature()``
+  when called on C-extention callables in tab completer.
+* Fixed issue that ``ls | less`` crashes on Mac.
+* Threadable prediction was incorrectly based on the user input command, rather than
+  the version where aliases have been resolved. This has been corrected.
+
+
+v0.5.4
+====================
+
+**Added:**
+
+* Add alias ``xip`` ("kip") so that xonsh's Python environment (whatever that is) can be modified.
+* HistoryEntry, a SimpleNamespace object that represents a command in history.
+* ``xonsh.completers.bash_completion`` module
+* Added option to report timing information of xonsh startup times. Start xonsh
+  with the ``--timings`` flag to use the feature.
+* The Python tab completer will now complete the argument names of functions
+  and other callables.
+* Uptime module added to ``xonsh.xoreutils``. This can report the system
+  boot time and up time.
+* The environment variable ``XONSH_HISTORY_BACKEND`` now also supports a
+  value of class type or a History Backend instance.
+* ``on_envvar_new`` event that fires after a new envvar is created.
+* ``on_envvar_change`` event that fires after an envvar is changed.
+
+
+**Changed:**
+
+* history indexing api to be more simple, now returns HistoryEntry.
+* Decoupled ``bash_completion`` from xonsh project and added shim back to
+  xonsh.
+* The JSON history backend will now unlock history files that were created
+  prior to the last reboot.
+
+
+**Fixed:**
+
+* Fixed broken bash completions on Windows if 'Windows Subsystem for Linux' is installed.
+* Readline history would try to read the first element of history prior to
+  actually loading any history. This caused an exception to be raised on
+  Windows at xonsh startup when using pyreadline.
+* Fixed issue with readline tab completer overwriting initial prefix in
+  some instances.
+* Fixed issue wherein if ``git`` or (presumably) ``hg`` are aliased, then branch
+  information no longer appears in the ``$PROMPT``
+* Fixed an issue with commands that background themselves (such as
+  ``gpg-connect-agent``) not being able to be run from within xonshrc.
+
+
+
+
 v0.5.3
 ====================
 

codecov.yml

@@ -0,0 +1,5 @@
+coverage:
+  status:
+    project:
+      default:
+        threshold: 1%

docs/api/index.rst

@@ -43,6 +43,7 @@ For those of you who want the gritty details.
     pretty
     replay
     diff_history
+    xoreutils/index
 
 
 **Helpers:**

docs/api/xoreutils/cat.rst

@@ -0,0 +1,10 @@
+.. _xonsh_xoreutils_cat:
+
+===============================================
+Cat Command -- :mod:`xonsh.xoreutils.cat`
+===============================================
+
+.. currentmodule:: xonsh.xoreutils.cat
+
+.. automodule:: xonsh.xoreutils.cat
+    :members:

docs/api/xoreutils/echo.rst

@@ -0,0 +1,10 @@
+.. _xonsh_xoreutils_echo:
+
+===============================================
+Echo Command -- :mod:`xonsh.xoreutils.echo`
+===============================================
+
+.. currentmodule:: xonsh.xoreutils.echo
+
+.. automodule:: xonsh.xoreutils.echo
+    :members:

docs/api/xoreutils/index.rst

@@ -0,0 +1,24 @@
+.. _api_xoreutils:
+
+==================
+Core Utilities API
+==================
+**Command Modules:**
+
+.. toctree::
+    :maxdepth: 1
+
+    cat
+    echo
+    pwd
+    tee
+    tty
+    which
+
+**Helper Modules:**
+
+.. toctree::
+    :maxdepth: 1
+
+    uptime
+    util

docs/api/xoreutils/pwd.rst

@@ -0,0 +1,10 @@
+.. _xonsh_xoreutils_pwd:
+
+===============================================
+Pwd Command -- :mod:`xonsh.xoreutils.pwd`
+===============================================
+
+.. currentmodule:: xonsh.xoreutils.pwd
+
+.. automodule:: xonsh.xoreutils.pwd
+    :members:

docs/api/xoreutils/tee.rst

@@ -0,0 +1,10 @@
+.. _xonsh_xoreutils_tee:
+
+===============================================
+Tee Command -- :mod:`xonsh.xoreutils.tee`
+===============================================
+
+.. currentmodule:: xonsh.xoreutils.tee
+
+.. automodule:: xonsh.xoreutils.tee
+    :members:

docs/api/xoreutils/tty.rst

@@ -0,0 +1,10 @@
+.. _xonsh_xoreutils_tty:
+
+===============================================
+TTY Command -- :mod:`xonsh.xoreutils.tty`
+===============================================
+
+.. currentmodule:: xonsh.xoreutils.tty
+
+.. automodule:: xonsh.xoreutils.tty
+    :members:

docs/api/xoreutils/uptime.rst

@@ -0,0 +1,10 @@
+.. _xonsh_xoreutils_uptime:
+
+===============================================
+System Uptime -- :mod:`xonsh.xoreutils.uptime`
+===============================================
+
+.. currentmodule:: xonsh.xoreutils.uptime
+
+.. automodule:: xonsh.xoreutils.uptime
+    :members:

docs/api/xoreutils/util.rst

@@ -0,0 +1,10 @@
+.. _xonsh_xoreutils_util:
+
+======================================================
+Core Utilites Utilities -- :mod:`xonsh.xoreutils.util`
+======================================================
+
+.. currentmodule:: xonsh.xoreutils.util
+
+.. automodule:: xonsh.xoreutils.util
+    :members:

docs/api/xoreutils/which.rst

@@ -0,0 +1,10 @@
+.. _xonsh_xoreutils_which:
+
+===============================================
+Which Command -- :mod:`xonsh.xoreutils.which`
+===============================================
+
+.. currentmodule:: xonsh.xoreutils.which
+
+.. automodule:: xonsh.xoreutils.which
+    :members:

docs/events.rst

@@ -1,5 +1,5 @@
 Core Events
 ===========
-The following events are defined by xonsh itself.
+The following events are defined by xonsh itself. For more information about events, see `the events tutorial <tutorial_events.html>`_.
 
 .. include:: eventsbody

docs/index.rst

@@ -131,6 +131,7 @@ Contents
     tutorial_events
     tutorial_completers
     tutorial_history_backend
+    tutorial_ptk
     bash_to_xsh
     python_virtual_environments
 

docs/tutorial.rst

@@ -1276,11 +1276,11 @@ Usually, callable alias commands will be run in a separate thread so that
 they may be run in the background.  However, some aliases may need to be
 executed on the thread that they were called from. This is mostly useful for
 debuggers and profilers. To make an alias run in the foreground, decorate its
-function with the ``xonsh.proc.unthreadable`` decorator.
+function with the ``xonsh.tools.unthreadable`` decorator.
 
 .. code-block:: python
 
-    from xonsh.proc import unthreadable
+    from xonsh.tools import unthreadable
 
     @unthreadable
     def _mycmd(args, stdin=None):
@@ -1297,12 +1297,12 @@ Thus the callable alias can't be captured without dire consequences (tm).
 To prevent this, you can declare a callable alias uncapturable. This is mostly
 useful for aliases that then open up text editors, pagers, or the like.
 To make an alias uncapturable, decorate its
-function with the ``xonsh.proc.uncapturable`` decorator. This is probably
+function with the ``xonsh.tools.uncapturable`` decorator. This is probably
 best used in conjunction with the ``unthreadable`` decorator.  For example:
 
 .. code-block:: python
 
-    from xonsh.proc import unthreadable, uncapturable
+    from xonsh.tools import unthreadable, uncapturable
 
     @uncapturable
     @unthreadable

docs/tutorial_events.rst

@@ -45,8 +45,10 @@ Yes! It's even easy! In your xontrib, you just have to do something like::
 
 This will enable users to call ``help(events.myxontrib_on_spam)`` and get useful output.
 
-Under the Hood
-==============
+Further Reading
+===============
+
+For a complete list of available events, see `the events reference <events.html>`_.
 
 If you want to know more about the gory details of what makes events tick, see 
 `Advanced Events <advanced_events.html>`_.

docs/tutorial_hist.rst

@@ -478,7 +478,7 @@ your ``~/.xonshrc`` file. To switch back to JSON version, remove this line,
 or set it to ``'json'``.
 
 .. note:: SQLite history backend currently only supports ``commands`` as
-the unit in ``$XONSH_HISTORY_SIZE`` in its garbage collection.
+    the unit in ``$XONSH_HISTORY_SIZE`` in its garbage collection.
 
 .. tip:: If you have `sqlite-web <https://pypi.python.org/pypi/sqlite-web>`_
     installed, you can read the history easily with command:

docs/tutorial_ptk.rst

@@ -0,0 +1,139 @@
+.. _tutorial_ptk:
+
+***********************************************
+Tutorial: ``prompt_toolkit`` custom keybindings
+***********************************************
+
+Are you really jonesing for some special keybindings? We can help you out with
+that. The first time is free and so is every other time!
+
+.. warning:: This tutorial will let you hook directly into the
+             ``prompt_toolkit`` keybinding manager. It will not stop you from
+             rendering your prompt completely unusable, so tread lightly.
+
+
+Overview
+========
+
+The ``prompt_toolkit`` shell has a registry for handling custom keybindings. You
+may not like the default keybindings in xonsh, or you may want to add a new key
+binding.
+
+We'll walk you though how to do this using ``prompt_toolkit`` tools to define
+keybindings and warn you about potential pitfalls.
+
+All of the code below can be entered into your `xonshrc <xonshrc.html>`_
+
+Control characters
+==================
+
+We can't and won't stop you from doing what you want, but in the interest of a
+functioning shell, you probably shouldn't mess with the following keystrokes.
+Some of them are `ASCII control characters
+<https://en.wikipedia.org/wiki/Control_character#In_ASCII>`_ and _really_
+shouldn't be used. The others are used by xonsh and will result in some loss of
+functionality (in less you take the time to rebind them elsewhere).
+
+.. list-table::
+    :widths: 2 2 2
+    :header-rows: 1
+
+    * - Keystroke
+      - ASCII control representation
+      - Default commmand
+    * - ``Control J``
+      - ``<Enter>``
+      - Run command
+    * - ``Control I``
+      - ``<Tab>``
+      - Indent, autocomplete
+    * - ``Control R``
+      - 
+      - Backwards history search
+    * - ``Control Z``
+      - 
+      - SIGSTOP current job
+    * - ``Control C``
+      -
+      - SIGINT current job
+
+
+Useful imports
+==============
+
+There are a few useful ``prompt_toolkit`` tools that will help us create better
+bindings::
+
+    from prompt_toolkit.keys import Keys
+    from prompt_toolkit.filters import Condition, EmacsInsertMode, ViInsertMode
+
+Custom keyload function
+=======================
+
+We need our additional keybindings to load after the shell is initialized, so we
+define a function that contains all of the custom keybindings and decorate it
+with the appropriate event, in this case ``on_ptk_create``.
+
+We'll start with a toy example that just inserts the text "hi" into the current line of the prompt::
+
+    @events.on_ptk_create
+    def custom_keybindings(bindings, **kw):
+        handler = bindings.registry.add_binding
+
+        @handler(Keys.ControlW)
+        def say_hi(event):
+            event.current_buffer.insert_text('hi')
+
+Put that in your `xonshrc <xonshrc.html>`_, restart xonsh and then see if
+pressing ``Ctrl-w`` does anything (it should!)
+
+What commands can keybindings run?
+==================================
+
+Pretty much anything! Since we're defining these commands after xonsh has
+started up, we can create keybinding events that run subprocess commands with
+hardly any effort at all. If we wanted to, say, have a command that runs ``ls
+-l`` in the current directory::
+
+    @handler(Keys.ControlP)
+    def run_ls(event):
+        ls -l
+        event.cli.renderer.erase()
+
+
+.. note:: The ``event.cli.renderer.erase()`` is required to redraw the prompt
+          after asking for a separate command to send information to ``STDOUT``
+
+Restrict actions with filters
+=============================
+
+Often we want a key command to only work if certain conditions are met. For
+instance, the ``<TAB>`` key in xonsh brings up the completions menu, but then it
+also cycles through the available completions. We use filters to create this
+behavior.
+
+A few helpful filters are included with ``prompt_toolkit``, like
+``ViInsertMode`` and ``EmacsInsertMode``, which return ``True`` when the
+respective insert mode is active.
+
+But it's also easy to create our own filters that take advantage of xonsh's
+beautiful strangeness. Suppose we want a filter to restrict a given command to
+run only when there are fewer than ten files in a given directory. We just need a function that returns a Bool that matches that requirement and then we decorate it! And remember, those functions can be in xonsh-language, not just pure Python::
+
+    @Condition
+    def lt_ten_files(cli):
+        return len(g`*`) < 10
+
+.. note:: See `the tutorial section on globbing
+          <tutorial.html#normal-globbing>`_ for more globbing options.
+
+Now that the condition is defined, we can pass it as a ``filter`` keyword to a keybinding definition::
+
+    @handler(Keys.ControlL, filter=lt_ten_files)
+    def ls_if_lt_ten(event):
+        ls -l
+        event.cli.renderer.erase()
+
+With both of those in your ``.xonshrc``, pressing ``Control L`` will list the
+contents of your current directory if there are fewer than 10 items in it.
+Useful? Debatable. Powerful? Yes.

news/argcomp.rst

@@ -1,14 +0,0 @@
-**Added:**
-
-* The Python tab completer will now complete the argument names of functions
-  and other callables.
-
-**Changed:** None
-
-**Deprecated:** None
-
-**Removed:** None
-
-**Fixed:** None
-
-**Security:** None

news/codecov.rst

@@ -1,6 +1,6 @@
 **Added:**
 
-* Add alias ``xip`` ("kip") so that xonsh's Python environment (whatever that is) can be modified.
+* CircleCI test post codecov run
 
 **Changed:** None
 

news/decoupled-bash-completion.rst

@@ -1,16 +0,0 @@
-**Added:**
-
-* ``xonsh.completers.bash_completion`` module
-
-**Changed:**
-
-* Decoupled ``bash_completion`` from xonsh project and added shim back to
-  xonsh.
-
-**Deprecated:** None
-
-**Removed:** None
-
-**Fixed:** None
-
-**Security:** None

news/event-envvar.rst

@@ -1,14 +0,0 @@
-**Added:**
-
-* ``on_envvar_new`` event that fires after a new envvar is created.
-* ``on_envvar_change`` event that fires after an envvar is changed.
-
-**Changed:** None
-
-**Deprecated:** None
-
-**Removed:** None
-
-**Fixed:** None
-
-**Security:** None

news/ghost-in-the-sh.rst

@@ -0,0 +1,14 @@
+**Added:** None
+
+**Changed:**
+
+* The prompt toolkit shell's first completion will now be the
+  current token from the auto-suggetion, if available.
+
+**Deprecated:** None
+
+**Removed:** None
+
+**Fixed:** None
+
+**Security:** None

news/give-term-to-none.rst

@@ -0,0 +1,14 @@
+**Added:** None
+
+**Changed:** None
+
+**Deprecated:** None
+
+**Removed:** None
+
+**Fixed:**
+
+* Fixed a potential "OSError: [Errno 22] Invalid argument" to increase job
+  control stability.
+
+**Security:** None

news/hist-be-env.rst

@@ -1,14 +0,0 @@
-**Added:**
-
-* The environment variable ``XONSH_HISTORY_BACKEND`` now also supports a
-  value of class type or a History Backend instance.
-
-**Changed:** None
-
-**Deprecated:** None
-
-**Removed:** None
-
-**Fixed:** None
-
-**Security:** None

news/history-indexing.rst

@@ -1,15 +0,0 @@
-**Added:**
-
-* HistoryEntry, a SimpleNamespace object that represents a command in history.
-
-**Changed:**
-
-* history indexing api to be more simple, now returns HistoryEntry.
-
-**Deprecated:** None
-
-**Removed:** None
-
-**Fixed:** None
-
-**Security:** None

news/ie.rst

@@ -0,0 +1,29 @@
+**Added:**
+
+* New events for hooking into the Python import process are now available.
+  You can now provide a handler for:
+
+  - ``on_import_pre_find_spec``
+  - ``on_import_post_find_spec``
+  - ``on_import_pre_create_module``
+  - ``on_import_post_create_module``
+  - ``on_import_pre_exec_module``
+  - ``on_import_post_exec_module``
+
+**Changed:** None
+
+**Deprecated:** None
+
+**Removed:** None
+
+**Fixed:**
+
+* The ``mpl`` xontrib has been updated to improve matplotlib
+  handling. If ``xontrib load mpl`` is run before matplotlib
+  is imported and xonsh is in ineteractive mode, matplotlib
+  will automatically enter interactive mode as well. Additionally,
+  ``pyplot.show()`` is patched in interactive mode to be non-blocking.
+  If a non-blocking show fails to draw the figre for some reason,
+  a regular blocking version is called.
+
+**Security:** None

news/insane.rst

@@ -1,14 +0,0 @@
-**Added:** None
-
-**Changed:** None
-
-**Deprecated:** None
-
-**Removed:** None
-
-**Fixed:**
-
-* Fixed an issue with commands that background themselves (such as
-  ``gpg-connect-agent``) not being able to be run from within xonshrc.
-
-**Security:** None

news/lex.rst

@@ -0,0 +1,17 @@
+**Added:**
+
+* The lexer has a new ``split()`` method which splits strings
+  according to xonsh's rules for whitespace and quotes.
+
+**Changed:** None
+
+**Deprecated:** None
+
+**Removed:** None
+
+**Fixed:**
+
+* The ``@$(cmd)`` operator now correctly splits strings according to
+  xonsh semantics, rather than just on whitespace using ``str.split()``.
+
+**Security:** None

news/linecont.rst

@@ -8,6 +8,7 @@
 
 **Fixed:**
 
-* Fixed broken bash completions on Windows if 'Windows Subsystem for Linux' is installed. 
+* Fixed issue with subprocess mode wrapping not repecting line continuation
+  backslashes.
 
 **Security:** None

news/lookup.rst

@@ -0,0 +1,14 @@
+**Added:** None
+
+**Changed:** None
+
+**Deprecated:** None
+
+**Removed:** None
+
+**Fixed:**
+
+* Threadable predicition for subprocesses will now consult both the command
+  as it was typed in and any resolved aliases.
+
+**Security:** None

news/no-fs-over.rst

@@ -0,0 +1,16 @@
+**Added:** None
+
+**Changed:**
+
+* Sourcing foreign shells will now safely skip applying aliases
+  with the same name as existing xonsh aliases by default.
+  This prevents accitidentally overwriting important xonsh standard
+  aliases, such as ``cd``.
+
+**Deprecated:** None
+
+**Removed:** None
+
+**Fixed:** None
+
+**Security:** None

news/rawpath.rst

@@ -0,0 +1,15 @@
+**Added:** None
+
+**Changed:** None
+
+**Deprecated:** None
+
+**Removed:** None
+
+**Fixed:**
+
+* Fixed issues pertaining to completing from raw string paths.
+  This is particularly relevant to Windows, where raw strings
+  are instered in path completion.
+
+**Security:** None

news/resetline.rst

@@ -0,0 +1,14 @@
+**Added:** None
+
+**Changed:** None
+
+**Deprecated:** None
+
+**Removed:** None
+
+**Fixed:**
+
+* The first prompt will no longer print in the middle of the line if the user has
+  already started typing.
+
+**Security:** None

news/rllen.rst

@@ -1,15 +0,0 @@
-**Added:** None
-
-**Changed:** None
-
-**Deprecated:** None
-
-**Removed:** None
-
-**Fixed:**
-
-* Readline history would try to read the first element of history prior to 
-  actually loading any history. This caused an exception to be raised on 
-  Windows at xonsh startup when using pyreadline.
-
-**Security:** None

news/term-fd.rst

@@ -8,7 +8,7 @@
 
 **Fixed:**
 
-* Fixed issue with readline tab completer overwriting initial prefix in
-  some instances.
+* Fixed issues like ``timeit ls`` causing OSError - "Inappropriate ioctl
+  for device".
 
 **Security:** None

news/tracer.rst

@@ -0,0 +1,17 @@
+**Added:**
+
+* The ``trace`` will automatically disable color printing when
+  stdout is not a TTY or stdout is captured.
+
+**Changed:** None
+
+**Deprecated:** None
+
+**Removed:** None
+
+**Fixed:**
+
+* The ``trace`` utility will now correctly color output and not
+  print extraneous newlines when called in a script.
+
+**Security:** None

news/uptime.rst

@@ -1,17 +0,0 @@
-**Added:**
-
-* Uptime module added to ``xonsh.xoreutils``. This can report the system
-  boot time and up time.
-
-**Changed:**
-
-* The JSON history backend will now unlock history files that were created
-  prior to the last reboot.
-
-**Deprecated:** None
-
-**Removed:** None
-
-**Fixed:** None
-
-**Security:** None

news/vc_alias_fix.rst

@@ -1,14 +0,0 @@
-**Added:** None
-
-**Changed:** None
-
-**Deprecated:** None
-
-**Removed:** None
-
-**Fixed:**
-
-* Fixed issue wherein if ``git`` or (presumably) ``hg`` are aliased, then branch
-  information no longer appears in the ``$PROMPT``
-
-**Security:** None

news/winterm.rst

@@ -0,0 +1,17 @@
+**Added:** None
+
+**Changed:** None
+
+**Deprecated:** None
+
+**Removed:** None
+
+**Fixed:**
+
+* Windows consoles will now automatically enable virtual terminal processing
+  with the readline shell, if available. This allows the full use of ANSI
+  escape sequences.
+* On the Windows readline shell, the teb-completion supression prompt will no
+  longer error out depending on what you press.
+
+**Security:** None

news/wsl_bash.rst

@@ -0,0 +1,14 @@
+**Added:** None
+
+**Changed:** None
+
+**Deprecated:** None
+
+**Removed:** None
+
+**Fixed:** 
+
+* Handle a bug where Bash On Windows causes platform.windows_bash_command() 
+  to raise CalledProcessError.
+
+**Security:** None

news/xoreutils.rst

@@ -0,0 +1,16 @@
+**Added:**
+
+* New core utility function aliases (written in pure Python) are now
+  available in ``xonsh.xoreutils``. These include: ``cat``, ``echo``,
+  ``pwd``, ``tee``, ``tty``, and ``yes``. These are not enabled by default.
+  Use the new ``coreutils`` xontrib to load them.
+
+**Changed:** None
+
+**Deprecated:** None
+
+**Removed:** None
+
+**Fixed:** None
+
+**Security:** None

tests/bin/printfile.xsh

@@ -0,0 +1,3 @@
+#!/usr/bin/env xonsh
+import os
+print(os.path.basename(__file__))

tests/bin/printname.xsh

@@ -0,0 +1,2 @@
+#!/usr/bin/env xonsh
+print(__name__)

tests/bin/sourcefile.xsh

@@ -0,0 +1,4 @@
+#!/usr/bin/env xonsh
+import os
+x = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'printfile.xsh')
+source @(x)

tests/conftest.py

@@ -4,8 +4,6 @@ import os
 
 import pytest
 
-import xonsh.built_ins
-
 from xonsh.built_ins import ensure_list_of_strs, enter_macro
 from xonsh.execer import Execer
 from xonsh.jobs import tasks
@@ -25,9 +23,9 @@ def source_path():
 @pytest.fixture
 def xonsh_execer(monkeypatch):
     """Initiate the Execer with a mocked nop `load_builtins`"""
-    monkeypatch.setattr(xonsh.built_ins, 'load_builtins',
-                        lambda *args, **kwargs: None)
-    execer = Execer(login=False, unload=False)
+    monkeypatch.setattr('xonsh.built_ins.load_builtins.__code__',
+                        (lambda *args, **kwargs: None).__code__)
+    execer = Execer(unload=False)
     builtins.__xonsh_execer__ = execer
     return execer
 
@@ -74,23 +72,34 @@ def xonsh_builtins(xonsh_events):
     # be firing events on the global instance.
     builtins.events = xonsh_events
     yield builtins
-    del builtins.__xonsh_env__
-    del builtins.__xonsh_ctx__
+    if hasattr(builtins, '__xonsh_env__'):
+        del builtins.__xonsh_env__
+    if hasattr(builtins, '__xonsh_ctx__'):
+        del builtins.__xonsh_ctx__
     del builtins.__xonsh_shell__
-    del builtins.__xonsh_help__
-    del builtins.__xonsh_glob__
-    del builtins.__xonsh_exit__
-    del builtins.__xonsh_superhelp__
+    if hasattr(builtins, '__xonsh_help__'):
+        del builtins.__xonsh_help__
+    if hasattr(builtins, '__xonsh_glob__'):
+        del builtins.__xonsh_glob__
+    if hasattr(builtins, '__xonsh_exit__'):
+        del builtins.__xonsh_exit__
+    if hasattr(builtins, '__xonsh_superhelp__'):
+        del builtins.__xonsh_superhelp__
     del builtins.__xonsh_regexpath__
-    del builtins.__xonsh_expand_path__
-    del builtins.__xonsh_stdout_uncaptured__
-    del builtins.__xonsh_stderr_uncaptured__
+    if hasattr(builtins, '__xonsh_expand_path__'):
+        del builtins.__xonsh_expand_path__
+    if hasattr(builtins, '__xonsh_stdout_uncaptured__'):
+        del builtins.__xonsh_stdout_uncaptured__
+    if hasattr(builtins, '__xonsh_stderr_uncaptured__'):
+        del builtins.__xonsh_stderr_uncaptured__
     del builtins.__xonsh_subproc_captured__
-    del builtins.__xonsh_subproc_uncaptured__
+    if hasattr(builtins, '__xonsh_subproc_uncaptured__'):
+        del builtins.__xonsh_subproc_uncaptured__
     del builtins.__xonsh_ensure_list_of_strs__
     del builtins.__xonsh_commands_cache__
     del builtins.__xonsh_all_jobs__
-    del builtins.__xonsh_history__
+    if hasattr(builtins, '__xonsh_history__'):
+        del builtins.__xonsh_history__
     del builtins.__xonsh_enter_macro__
     del builtins.evalx
     del builtins.execx

tests/test_execer.py

@@ -104,3 +104,8 @@ def test_echo_comma_val():
 def test_echo_comma_2val():
     code = 'echo 1,2\n'
     assert check_parse(code)
+
+
+def test_echo_line_cont():
+    code = 'echo "1 \\\n2"\n'
+    assert check_parse(code)

tests/test_imphooks.py

@@ -6,16 +6,16 @@ import builtins
 import pytest
 
 from xonsh import imphooks
+from xonsh.execer import Execer
 from xonsh.environ import Env
-from xonsh.built_ins import load_builtins, unload_builtins
+from xonsh.built_ins import unload_builtins
 
-imphooks.install_hook()
+imphooks.install_import_hooks()
 
 
 @pytest.yield_fixture(autouse=True)
-def imp_env(xonsh_execer):
-    """Call `load_builtins` with `xonsh_execer`"""
-    load_builtins(execer=xonsh_execer)
+def imp_env():
+    execer = Execer(unload=False)
     builtins.__xonsh_env__ = Env({'PATH': [], 'PATHEXT': []})
     yield
     unload_builtins()

tests/test_integrations.py

@@ -8,7 +8,8 @@ import pytest
 import xonsh
 from xonsh.platform import ON_WINDOWS
 
-from tools import skip_if_on_windows
+from tools import (skip_if_on_windows, skip_if_on_darwin, skip_if_on_travis,
+    ON_WINDOWS, ON_DARWIN, ON_TRAVIS)
 
 
 XONSH_PREFIX = xonsh.__file__
@@ -28,6 +29,10 @@ PATH = os.path.join(os.path.dirname(__file__), 'bin') + os.pathsep + \
        os.environ['PATH']
 
 
+skip_if_no_xonsh = pytest.mark.skipif(shutil.which('xonsh', path=PATH) is None,
+                                      reason='xonsh not on path')
+
+
 def run_xonsh(cmd, stdin=sp.PIPE, stdout=sp.PIPE, stderr=sp.STDOUT):
     env = dict(os.environ)
     env['PATH'] = PATH
@@ -51,6 +56,20 @@ def run_xonsh(cmd, stdin=sp.PIPE, stdout=sp.PIPE, stderr=sp.STDOUT):
         raise
     return out, err, proc.returncode
 
+
+def check_run_xonsh(cmd, fmt, exp):
+    """The ``fmt`` parameter is a function
+    that formats the output of cmd, can be None.
+    """
+    out, err, rtn = run_xonsh(cmd, stderr=sp.DEVNULL)
+    if callable(fmt):
+        out = fmt(out)
+    if callable(exp):
+        exp = exp()
+    assert out == exp
+    assert rtn == 0
+
+
 #
 # The following list contains a (stdin, stdout, returncode) tuples
 #
@@ -146,6 +165,17 @@ with open('tttt', 'w') as fp:
 
 ![cat tttt | wc | wc]
 """, '      1       3      24\n' if ON_WINDOWS else " 1  4 16 <stdin>\n", 0),
+# test unthreadable alias (which should trigger a ProcPoxy call)
+("""
+from xonsh.tools import unthreadable
+
+@unthreadable
+def _f():
+    return 'hello\\n'
+
+aliases['f'] = _f
+f
+""", "hello\n", 0),
 ]
 
 
@@ -175,24 +205,33 @@ def test_script_stder(case):
     assert exp_err == err
     assert exp_rtn == rtn
 
-
 @skip_if_on_windows
 @pytest.mark.parametrize('cmd, fmt, exp', [
     ('pwd', None, lambda: os.getcwd() + '\n'),
     ('echo WORKING', None, 'WORKING\n'),
     ('ls -f', lambda out: out.splitlines().sort(), os.listdir().sort()),
     ])
-def test_single_command(cmd, fmt, exp):
-    """The ``fmt`` parameter is a function
-    that formats the output of cmd, can be None.
-    """
-    out, err, rtn = run_xonsh(cmd, stderr=sp.DEVNULL)
-    if callable(fmt):
-        out = fmt(out)
-    if callable(exp):
-        exp = exp()
-    assert out == exp
-    assert rtn == 0
+def test_single_command_no_windows(cmd, fmt, exp):
+    check_run_xonsh(cmd, fmt, exp)
+
+
+_bad_case = pytest.mark.skipif(ON_DARWIN or ON_WINDOWS or ON_TRAVIS,
+                               reason="bad platforms")
+
+@_bad_case
+def test_printfile():
+    check_run_xonsh('printfile.xsh', None, 'printfile.xsh\n')
+
+
+@_bad_case
+def test_printname():
+    check_run_xonsh('printfile.xsh', None, 'printfile.xsh\n')
+
+
+@_bad_case
+def test_sourcefile():
+    check_run_xonsh('printfile.xsh', None, 'printfile.xsh\n')
+
 
 
 @skip_if_on_windows

tests/test_lexer.py

@@ -32,12 +32,14 @@ def ensure_tuple(x):
         raise TypeError('{0} is not a sequence'.format(x))
     return x
 
+
 def tokens_equal(x, y):
     """Tests whether two token are equal."""
     xtup = ensure_tuple(x)
     ytup = ensure_tuple(y)
     return xtup == ytup
 
+
 def assert_token_equal(x, y):
     """Asserts that two tokens are equal."""
     if not tokens_equal(x, y):
@@ -45,6 +47,7 @@ def assert_token_equal(x, y):
         pytest.fail(msg)
     return True
 
+
 def assert_tokens_equal(x, y):
     """Asserts that two token sequences are equal."""
     if len(x) != len(y):
@@ -60,6 +63,7 @@ def assert_tokens_equal(x, y):
         pytest.fail(msg)
     return True
 
+
 def check_token(inp, exp):
     l = Lexer()
     l.input(inp)
@@ -70,41 +74,50 @@ def check_token(inp, exp):
         pytest.fail(msg.format(len(obs), pformat(obs)))
     return assert_token_equal(exp, obs[0])
 
+
 def check_tokens(inp, exp):
     l = Lexer()
     l.input(inp)
     obs = list(l)
     return assert_tokens_equal(exp, obs)
 
+
 def check_tokens_subproc(inp, exp):
     l = Lexer()
     l.input('$[{}]'.format(inp))
     obs = list(l)[1:-1]
     return assert_tokens_equal(exp, obs)
 
+
 def test_int_literal():
     assert check_token('42', ['NUMBER', '42', 0])
 
+
 def test_hex_literal():
     assert check_token('0x42', ['NUMBER', '0x42', 0])
 
+
 def test_oct_o_literal():
     assert check_token('0o42', ['NUMBER', '0o42', 0])
 
+
 def test_bin_literal():
     assert check_token('0b101010', ['NUMBER', '0b101010', 0])
 
+
 def test_indent():
     exp = [('INDENT', '  \t  ', 0),
            ('NUMBER', '42', 5),
            ('DEDENT', '', 0)]
     assert check_tokens('  \t  42', exp)
 
+
 def test_post_whitespace():
     inp = '42  \t  '
     exp = [('NUMBER', '42', 0)]
     assert check_tokens(inp, exp)
 
+
 def test_internal_whitespace():
     inp = '42  +\t65'
     exp = [('NUMBER', '42', 0),
@@ -112,6 +125,7 @@ def test_internal_whitespace():
            ('NUMBER', '65', 6),]
     assert check_tokens(inp, exp)
 
+
 def test_indent_internal_whitespace():
     inp = ' 42  +\t65'
     exp = [('INDENT', ' ', 0),
@@ -121,6 +135,7 @@ def test_indent_internal_whitespace():
            ('DEDENT', '', 0)]
     assert check_tokens(inp, exp)
 
+
 def test_assignment():
     inp = 'x = 42'
     exp = [('NAME', 'x', 0),
@@ -128,6 +143,7 @@ def test_assignment():
            ('NUMBER', '42', 4),]
     assert check_tokens(inp, exp)
 
+
 def test_multiline():
     inp = 'x\ny'
     exp = [('NAME', 'x', 0),
@@ -144,51 +160,67 @@ def test_atdollar_expression():
            ('RPAREN', ')', 15)]
     assert check_tokens(inp, exp)
 
+
 def test_and():
     assert check_token('and', ['AND', 'and', 0])
 
+
 def test_ampersand():
     assert check_token('&', ['AMPERSAND', '&', 0])
 
+
 def test_atdollar():
     assert check_token('@$', ['ATDOLLAR', '@$', 0])
 
+
 def test_doubleamp():
     assert check_token('&&', ['AND', 'and', 0])
 
+
 def test_pipe():
     assert check_token('|', ['PIPE', '|', 0])
 
+
 def test_doublepipe():
     assert check_token('||', ['OR', 'or', 0])
 
+
 def test_single_quote_literal():
     assert check_token("'yo'", ['STRING', "'yo'", 0])
 
+
 def test_double_quote_literal():
     assert check_token('"yo"', ['STRING', '"yo"', 0])
 
+
 def test_triple_single_quote_literal():
     assert check_token("'''yo'''", ['STRING', "'''yo'''", 0])
 
+
 def test_triple_double_quote_literal():
     assert check_token('"""yo"""', ['STRING', '"""yo"""', 0])
 
+
 def test_single_raw_string_literal():
     assert check_token("r'yo'", ['STRING', "r'yo'", 0])
 
+
 def test_double_raw_string_literal():
     assert check_token('r"yo"', ['STRING', 'r"yo"', 0])
 
+
 def test_single_unicode_literal():
     assert check_token("u'yo'", ['STRING', "u'yo'", 0])
 
+
 def test_double_unicode_literal():
     assert check_token('u"yo"', ['STRING', 'u"yo"', 0])
 
+
 def test_single_bytes_literal():
     assert check_token("b'yo'", ['STRING', "b'yo'", 0])
 
+
 def test_path_string_literal():
     assert check_token("p'/foo'", ['STRING', "p'/foo'", 0])
     assert check_token('p"/foo"', ['STRING', 'p"/foo"', 0])
@@ -204,12 +236,36 @@ def test_regex_globs():
             c = '{}`{}`'.format(p,i)
             assert check_token(c, ['SEARCHPATH', c, 0])
 
+
 @pytest.mark.parametrize('case', [
     '0.0', '.0', '0.', '1e10', '1.e42', '0.1e42', '0.5e-42', '5E10', '5e+42'])
 def test_float_literals(case):
     assert check_token(case, ['NUMBER', case, 0])
 
+
 def test_ioredir():
     cases = ['2>1', 'err>out', 'o>', 'all>', 'e>o', 'e>', 'out>', '2>&1']
     for s in cases:
         assert check_tokens_subproc(s, [('IOREDIRECT', s, 2)])
+
+
+@pytest.mark.parametrize('s, exp', [
+    ('', []),
+    ('   \t   \n \t  ', []),
+    ('echo hello', ['echo', 'hello']),
+    ('echo "hello"', ['echo', '"hello"']),
+    ('![echo "hello"]', ['![echo', '"hello"]']),
+    ('/usr/bin/echo hello', ['/usr/bin/echo', 'hello']),
+    ('$(/usr/bin/echo hello)', ['$(/usr/bin/echo', 'hello)']),
+    ('C:\\Python\\python.exe -m xonsh', ['C:\\Python\\python.exe', '-m', 'xonsh']),
+    ('print("""I am a triple string""")', ['print("""I am a triple string""")']),
+    ('print("""I am a \ntriple string""")', ['print("""I am a \ntriple string""")']),
+    ('echo $HOME', ['echo', '$HOME']),
+    ('echo -n $HOME', ['echo', '-n', '$HOME']),
+    ('echo --go=away', ['echo', '--go=away']),
+    ('echo --go=$HOME', ['echo', '--go=$HOME']),
+])
+def test_lexer_split(s, exp):
+    lexer = Lexer()
+    obs = lexer.split(s)
+    assert exp == obs

tests/test_main.py

@@ -17,7 +17,7 @@ def Shell(*args, **kwargs):
 
 
 @pytest.fixture
-def shell(xonsh_builtins, monkeypatch):
+def shell(xonsh_builtins, xonsh_execer, monkeypatch):
     """Xonsh Shell Mock"""
     monkeypatch.setattr(xonsh.main, 'Shell', Shell)
 
@@ -62,7 +62,7 @@ def test_premain_interactive__with_file_argument(shell):
 
 
 @pytest.mark.parametrize('case', ['----', '--hep', '-TT', '--TTTT'])
-def test_premain_invalid_arguments(case, shell, capsys):
+def test_premain_invalid_arguments(shell, case, capsys):
     with pytest.raises(SystemExit):
         xonsh.main.premain([case])
     assert 'unrecognized argument' in capsys.readouterr()[1]

tests/test_replay.py

@@ -7,6 +7,7 @@ import builtins
 import pytest
 
 from xonsh.shell import Shell
+from xonsh.execer import Execer
 from xonsh.replay import Replayer
 
 from tools import skip_if_on_darwin
@@ -18,7 +19,9 @@ HISTDIR = os.path.join(os.path.dirname(__file__), 'histories')
 @pytest.yield_fixture(scope='module', autouse=True)
 def ctx():
     """Create a global Shell instance to use in all the test."""
-    builtins.__xonsh_shell__ = Shell({'PATH': []})
+    ctx = {'PATH': []}
+    execer = Execer(xonsh_ctx=ctx)
+    builtins.__xonsh_shell__ = Shell(execer=execer, ctx=ctx)
     yield
     del builtins.__xonsh_shell__
 

tests/test_tools.py

@@ -23,7 +23,8 @@ from xonsh.tools import (
     to_dynamic_cwd_tuple, to_logfile_opt, pathsep_to_set, set_to_pathsep,
     is_string_seq, pathsep_to_seq, seq_to_pathsep, is_nonstring_seq_of_strings,
     pathsep_to_upper_seq, seq_to_upper_pathsep, expandvars, is_int_as_str, is_slice_as_str,
-    ensure_timestamp, get_portions, is_balanced, subexpr_before_unbalanced
+    ensure_timestamp, get_portions, is_balanced, subexpr_before_unbalanced,
+    swap_values, get_logical_line, replace_logical_line
     )
 from xonsh.environ import Env
 
@@ -322,6 +323,36 @@ def test_subproc_toks_pyeval_redirect():
     assert exp == obs
 
 
+LOGICAL_LINE_CASES = [
+("""x = 14 + 2""", 0, 'x = 14 + 2', 1),
+("""x = \\
+14 \\
++ 2
+""", 0, 'x = 14 + 2', 3),
+("""y = 16
+14 \\
++ 2
+""", 1, '14 + 2', 2),
+]
+
+@pytest.mark.parametrize('src, idx, exp_line, exp_n', LOGICAL_LINE_CASES)
+def test_get_logical_line(src, idx, exp_line, exp_n):
+    lines = src.splitlines()
+    line, n = get_logical_line(lines, idx)
+    assert exp_line == line
+    assert exp_n == n
+
+
+@pytest.mark.parametrize('src, idx, exp_line, exp_n', LOGICAL_LINE_CASES)
+def test_replace_logical_line(src, idx, exp_line, exp_n):
+    lines = src.splitlines()
+    logical = exp_line
+    replace_logical_line(lines, logical, idx, exp_n)
+    exp = src.replace('\\\n', '').strip()
+    obs = '\n'.join(lines).replace('\\\n', '').strip()
+    assert exp == obs
+
+
 @pytest.mark.parametrize('inp', [
     'f(1,10),x.y',
 ])
@@ -1128,3 +1159,13 @@ def test_expand_path(expand_user, inp, expand_env_vars, exp_end, xonsh_builtins)
         assert path == home_path + exp_end
     else:
         assert path == '~' + exp_end
+
+
+def test_swap_values():
+    orig = {'x': 1}
+    updates = {'x': 42, 'y': 43}
+    with swap_values(orig, updates):
+        assert orig['x'] == 42
+        assert orig['y'] == 43
+    assert orig['x'] == 1
+    assert 'y' not in orig

tests/test_xonsh.xsh

@@ -1,3 +1,6 @@
+import builtins
+
+
 def test_simple():
   assert 1 + 1 == 2
 
@@ -10,7 +13,12 @@ def test_envionment():
 
 
 def test_xonsh_party():
-  x = 'xonsh'
-  y = 'party'
-  out = $(echo @(x + '-' + y)).strip()
-  assert out == 'xonsh-party', 'Out really was <' + out + '>, sorry.'
+  orig = builtins.__xonsh_env__.get('XONSH_INTERACTIVE')
+  builtins.__xonsh_env__['XONSH_INTERACTIVE'] = False
+  try:
+      x = 'xonsh'
+      y = 'party'
+      out = $(echo @(x + '-' + y)).strip()
+      assert out == 'xonsh-party', 'Out really was <' + out + '>, sorry.'
+  finally:
+      builtins.__xonsh_env__['XONSH_INTERACTIVE'] = orig

tests/tools.py

@@ -23,8 +23,9 @@ VER_MAJOR_MINOR = sys.version_info[:2]
 VER_FULL = sys.version_info[:3]
 ON_DARWIN = (platform.system() == 'Darwin')
 ON_WINDOWS = (platform.system() == 'Windows')
-ON_CONDA = True in [conda in pytest.__file__ for conda
+ON_CONDA = True in [conda in pytest.__file__.lower() for conda
                     in ['anaconda', 'miniconda']]
+ON_TRAVIS = 'TRAVIS' in os.environ and 'CI' in os.environ
 TEST_DIR = os.path.dirname(__file__)
 
 # pytest skip decorators
@@ -39,13 +40,15 @@ skip_if_on_unix = pytest.mark.skipif(not ON_WINDOWS, reason='Windows stuff')
 
 skip_if_on_darwin = pytest.mark.skipif(ON_DARWIN, reason='not Mac friendly')
 
+skip_if_on_travis = pytest.mark.skipif(ON_TRAVIS, reason='not Travis CI friendly')
+
 
 def sp(cmd):
     return subprocess.check_output(cmd, universal_newlines=True)
 
 
 class DummyStyler():
-    styles = defaultdict(None.__class__)
+    styles = defaultdict(str)
 
 
 class DummyBaseShell(BaseShell):
@@ -89,6 +92,7 @@ class DummyEnv(MutableMapping):
 
     DEFAULTS = {
         'XONSH_DEBUG': 1,
+        'XONSH_COLOR_STYLE': 'default',
     }
 
     def __init__(self, *args, **kwargs):

xonsh/__init__.py

@@ -1,4 +1,5 @@
-__version__ = '0.5.3'
+__version__ = '0.5.5'
+
 
 # amalgamate exclude jupyter_kernel parser_table parser_test_table pyghooks
 # amalgamate exclude winutils wizard pytest_plugin fs macutils
@@ -25,10 +26,6 @@ else:
         _sys.modules['xonsh.platform'] = __amalgam__
         pretty = __amalgam__
         _sys.modules['xonsh.pretty'] = __amalgam__
-        timings = __amalgam__
-        _sys.modules['xonsh.timings'] = __amalgam__
-        jobs = __amalgam__
-        _sys.modules['xonsh.jobs'] = __amalgam__
         lazyimps = __amalgam__
         _sys.modules['xonsh.lazyimps'] = __amalgam__
         parser = __amalgam__
@@ -49,24 +46,34 @@ else:
         _sys.modules['xonsh.events'] = __amalgam__
         foreign_shells = __amalgam__
         _sys.modules['xonsh.foreign_shells'] = __amalgam__
+        jobs = __amalgam__
+        _sys.modules['xonsh.jobs'] = __amalgam__
         lexer = __amalgam__
         _sys.modules['xonsh.lexer'] = __amalgam__
         openpy = __amalgam__
         _sys.modules['xonsh.openpy'] = __amalgam__
-        proc = __amalgam__
-        _sys.modules['xonsh.proc'] = __amalgam__
         xontribs = __amalgam__
         _sys.modules['xonsh.xontribs'] = __amalgam__
         dirstack = __amalgam__
         _sys.modules['xonsh.dirstack'] = __amalgam__
         inspectors = __amalgam__
         _sys.modules['xonsh.inspectors'] = __amalgam__
+        proc = __amalgam__
+        _sys.modules['xonsh.proc'] = __amalgam__
+        shell = __amalgam__
+        _sys.modules['xonsh.shell'] = __amalgam__
+        timings = __amalgam__
+        _sys.modules['xonsh.timings'] = __amalgam__
         xonfig = __amalgam__
         _sys.modules['xonsh.xonfig'] = __amalgam__
+        base_shell = __amalgam__
+        _sys.modules['xonsh.base_shell'] = __amalgam__
         environ = __amalgam__
         _sys.modules['xonsh.environ'] = __amalgam__
         tracer = __amalgam__
         _sys.modules['xonsh.tracer'] = __amalgam__
+        readline_shell = __amalgam__
+        _sys.modules['xonsh.readline_shell'] = __amalgam__
         replay = __amalgam__
         _sys.modules['xonsh.replay'] = __amalgam__
         aliases = __amalgam__
@@ -77,14 +84,8 @@ else:
         _sys.modules['xonsh.execer'] = __amalgam__
         imphooks = __amalgam__
         _sys.modules['xonsh.imphooks'] = __amalgam__
-        shell = __amalgam__
-        _sys.modules['xonsh.shell'] = __amalgam__
-        base_shell = __amalgam__
-        _sys.modules['xonsh.base_shell'] = __amalgam__
         main = __amalgam__
         _sys.modules['xonsh.main'] = __amalgam__
-        readline_shell = __amalgam__
-        _sys.modules['xonsh.readline_shell'] = __amalgam__
         del __amalgam__
     except ImportError:
         pass

xonsh/aliases.py

@@ -15,10 +15,10 @@ from xonsh.foreign_shells import foreign_shell_data
 from xonsh.jobs import jobs, fg, bg, clean_jobs
 from xonsh.platform import (ON_ANACONDA, ON_DARWIN, ON_WINDOWS, ON_FREEBSD,
                             ON_NETBSD)
-from xonsh.proc import foreground
+from xonsh.tools import unthreadable
 from xonsh.replay import replay_main
 from xonsh.timings import timeit_alias
-from xonsh.tools import argvquote, escape_windows_cmd_string, to_bool
+from xonsh.tools import argvquote, escape_windows_cmd_string, to_bool, swap_values
 from xonsh.xontribs import xontribs_main
 
 import xonsh.completers._aliases as xca
@@ -199,10 +199,14 @@ def _SOURCE_FOREIGN_PARSER():
     parser.add_argument('--seterrpostcmd', default=None, dest='seterrpostcmd',
                         help='command(s) to set exit-on-error after all'
                              'other commands.')
+    parser.add_argument('--overwrite-aliases', default=False, action='store_true',
+                        dest='overwrite_aliases',
+                        help='flag for whether or not sourced aliases should '
+                             'replace the current xonsh aliases.')
     return parser
 
 
-def source_foreign(args, stdin=None):
+def source_foreign(args, stdin=None, stdout=None, stderr=None):
     """Sources a file written in a foreign shell language."""
     ns = _SOURCE_FOREIGN_PARSER.parse_args(args)
     if ns.prevcmd is not None:
@@ -244,7 +248,13 @@ def source_foreign(args, stdin=None):
     for k, v in fsaliases.items():
         if k in baliases and v == baliases[k]:
             continue  # no change from original
-        baliases[k] = v
+        elif ns.overwite_aliases or k not in baliases:
+            baliases[k] = v
+        else:
+            msg = ('Skipping application of {0!r} alias from {1!r} '
+                   'since it shares a name with an existing xonsh alias. '
+                   'Use "--overwrite-alias" option to apply it anyway.')
+            print(msg.format(k, ns.shell), file=stderr)
 
 
 def source_alias(args, stdin=None):
@@ -270,8 +280,10 @@ def source_alias(args, stdin=None):
             src = fp.read()
         if not src.endswith('\n'):
             src += '\n'
-        with env.swap(ARGS=args[i+1:]):
-            builtins.execx(src, 'exec', builtins.__xonsh_ctx__, filename=fpath)
+        ctx = builtins.__xonsh_ctx__
+        updates = {'__file__': fpath, '__name__': os.path.abspath(fpath)}
+        with env.swap(ARGS=args[i+1:]), swap_values(ctx, updates):
+            builtins.execx(src, 'exec', ctx, filename=fpath)
 
 
 def source_cmd(args, stdin=None):
@@ -350,12 +362,13 @@ def xonfig(args, stdin=None):
     return xonfig_main(args)
 
 
-@foreground
-def trace(args, stdin=None):
+@unthreadable
+def trace(args, stdin=None, stdout=None, stderr=None, spec=None):
     """Runs the xonsh tracer utility."""
     from xonsh.tracer import tracermain  # lazy import
     try:
-        return tracermain(args)
+        return tracermain(args, stdin=stdin, stdout=stdout,
+                          stderr=stderr, spec=spec)
     except SystemExit:
         pass
 

xonsh/base_shell.py

@@ -16,6 +16,8 @@ from xonsh.completer import Completer
 from xonsh.prompt.base import multiline_prompt, PromptFormatter
 from xonsh.events import events
 from xonsh.shell import transform_command
+from xonsh.lazyimps import pygments, pyghooks
+from xonsh.ansi_colors import ansi_partial_color_format
 
 if ON_WINDOWS:
     import ctypes
@@ -402,6 +404,9 @@ class BaseShell(object):
             if usecache:
                 self.reset_buffer()
                 return src, code
+        if src.endswith('\\\n'):
+            self.need_more_lines = True
+            return src, None
         try:
             code = self.execer.compile(src,
                                        mode='single',
@@ -474,19 +479,32 @@ class BaseShell(object):
         self.settitle()
         return p
 
-    def format_color(self, string, **kwargs):
-        """Formats the colors in a string. This base implmentation does not
-        actually do any coloring, but just returns the string directly.
+    def format_color(self, string, hide=False, force_string=False, **kwargs):
+        """Formats the colors in a string. This base implmentation colors based
+        on ANSI color codes
         """
-        return string
+        style = builtins.__xonsh_env__.get('XONSH_COLOR_STYLE')
+        return ansi_partial_color_format(string, hide=hide, style=style)
 
-    def print_color(self, string, **kwargs):
-        """Prints a string in color. This base implmentation does not actually
-        do any coloring, but just prints the string directly.
+    def print_color(self, string, hide=False, **kwargs):
+        """Prints a string in color. This base implmentation will color based
+        ANSI color codes if a string was given as input. If a list of token
+        pairs is given, it will color based on pygments, if available. If
+        pygments is not available, it will print a colorless string.
         """
-        if not isinstance(string, str):
-            string = ''.join([x for _, x in string])
-        print(string, **kwargs)
+        if isinstance(string, str):
+            s = self.format_color(string, hide=hide)
+        elif HAS_PYGMENTS:
+            # assume this is a list of (Token, str) tuples and format it
+            env = builtins.__xonsh_env__
+            self.styler.style_name = env.get('XONSH_COLOR_STYLE')
+            style_proxy = pyghooks.xonsh_style_proxy(self.styler)
+            formatter = pyghooks.XonshTerminal256Formatter(style=style_proxy)
+            s = pygments.format(string, formatter).rstrip()
+        else:
+            # assume this is a list of (Token, str) tuples and remove color
+            s = ''.join([x for _, x in string])
+        print(s, **kwargs)
 
     def color_style_names(self):
         """Returns an iterable of all available style names."""

xonsh/built_ins.py

@@ -8,7 +8,6 @@ import io
 import os
 import re
 import sys
-import time
 import types
 import shlex
 import signal
@@ -26,7 +25,6 @@ from xonsh.lazyasd import LazyObject, lazyobject
 from xonsh.inspectors import Inspector
 from xonsh.aliases import Aliases, make_default_aliases
 from xonsh.environ import Env, default_env, locate_binary
-from xonsh.foreign_shells import load_foreign_aliases
 from xonsh.jobs import add_job
 from xonsh.platform import ON_POSIX, ON_WINDOWS
 from xonsh.proc import (
@@ -283,7 +281,7 @@ def safe_close(x):
         pass
 
 
-def _parse_redirects(r):
+def _parse_redirects(r, loc=None):
     """returns origin, mode, destination tuple"""
     orig, mode, dest = _REDIR_REGEX.match(r).groups()
     # redirect to fd
@@ -496,6 +494,8 @@ class SubprocSpec:
         self.prep_env(kwargs)
         self.prep_preexec_fn(kwargs, pipeline_group=pipeline_group)
         if callable(self.alias):
+            if 'preexec_fn' in kwargs:
+                kwargs.pop('preexec_fn')
             p = self.cls(self.alias, self.cmd, **kwargs)
         else:
             p = self._run_binary(kwargs)
@@ -533,7 +533,9 @@ class SubprocSpec:
 
     def prep_preexec_fn(self, kwargs, pipeline_group=None):
         """Prepares the 'preexec_fn' keyword argument"""
-        if not (ON_POSIX and self.cls is subprocess.Popen):
+        if not ON_POSIX:
+            return
+        if not builtins.__xonsh_env__.get('XONSH_INTERACTIVE'):
             return
         if pipeline_group is None:
             xonsh_preexec_fn = no_pg_xonsh_preexec_fn
@@ -674,13 +676,17 @@ def _update_last_spec(last):
     if callable_alias:
         pass
     else:
-        thable = builtins.__xonsh_commands_cache__.predict_threadable(last.args)
+        cmds_cache = builtins.__xonsh_commands_cache__
+        thable = (cmds_cache.predict_threadable(last.args) and
+                  cmds_cache.predict_threadable(last.cmd))
         if captured and thable:
             last.cls = PopenThread
         elif not thable:
-            # foreground processes should use Popen and not pipe stdout, stderr
+            # foreground processes should use Popen
             last.threadable = False
-            return
+            if captured == 'object' or captured == 'hiddenobject':
+                # CommandPipeline objects should not pipe stdout, stderr
+                return
     # cannot used PTY pipes for aliases, for some dark reason,
     # and must use normal pipes instead.
     use_tty = ON_POSIX and not callable_alias
@@ -789,34 +795,27 @@ def run_subproc(cmds, captured=False):
     """
     specs = cmds_to_specs(cmds, captured=captured)
     captured = specs[-1].captured
-    procs = []
-    proc = pipeline_group = None
-    for spec in specs:
-        starttime = time.time()
-        proc = spec.run(pipeline_group=pipeline_group)
-        procs.append(proc)
-        if ON_POSIX and pipeline_group is None and \
-           spec.cls is subprocess.Popen:
-            pipeline_group = proc.pid
-    if not spec.is_proxy:
+    if captured == 'hiddenobject':
+        command = HiddenCommandPipeline(specs)
+    else:
+        command = CommandPipeline(specs)
+    proc = command.proc
+    background = command.spec.background
+    if not all(x.is_proxy for x in specs):
         add_job({
             'cmds': cmds,
-            'pids': [i.pid for i in procs],
+            'pids': [i.pid for i in command.procs],
             'obj': proc,
-            'bg': spec.background,
+            'bg': background,
+            'pipeline': command,
+            'pgrp': command.term_pgid,
         })
     if _should_set_title(captured=captured):
         # set title here to get currently executing command
         pause_call_resume(proc, builtins.__xonsh_shell__.settitle)
     # create command or return if backgrounding.
-    if spec.background:
+    if background:
         return
-    if captured == 'hiddenobject':
-        command = HiddenCommandPipeline(specs, procs, starttime=starttime,
-                                        captured=captured)
-    else:
-        command = CommandPipeline(specs, procs, starttime=starttime,
-                                  captured=captured)
     # now figure out what we should return.
     if captured == 'stdout':
         command.end()
@@ -840,8 +839,13 @@ def subproc_captured_stdout(*cmds):
 
 def subproc_captured_inject(*cmds):
     """Runs a subprocess, capturing the output. Returns a list of
-    whitespace-separated strings in the stdout that was produced."""
-    return [i.strip() for i in run_subproc(cmds, captured='stdout').split()]
+    whitespace-separated strings of the stdout that was produced.
+    The string is split using xonsh's lexer, rather than Python's str.split()
+    or shlex.split().
+    """
+    s = run_subproc(cmds, captured='stdout')
+    toks = builtins.__xonsh_execer__.parser.lexer.split(s)
+    return toks
 
 
 def subproc_captured_object(*cmds):
@@ -905,7 +909,7 @@ def MACRO_FLAG_KINDS():
         'exec': exec,
         't': type,
         'type': type,
-        }
+    }
 
 
 def _convert_kind_flag(x):
@@ -958,7 +962,7 @@ def convert_macro_arg(raw_arg, kind, glbs, locs, *, name='<arg>',
             ctx |= set(locs.keys())
         mode = mode or 'eval'
         arg = execer.parse(raw_arg, ctx, mode=mode, filename=filename)
-    elif kind is types.CodeType or kind is compile:
+    elif kind is types.CodeType or kind is compile:  # NOQA
         mode = mode or 'eval'
         arg = execer.compile(raw_arg, mode=mode, glbs=glbs, locs=locs,
                              filename=filename)
@@ -1120,14 +1124,14 @@ def enter_macro(obj, raw_block, glbs, locs):
     return obj
 
 
-def load_builtins(execer=None, config=None, login=False, ctx=None):
+def load_builtins(execer=None, ctx=None):
     """Loads the xonsh builtins into the Python builtins. Sets the
     BUILTINS_LOADED variable to True.
     """
     global BUILTINS_LOADED
     # private built-ins
     builtins.__xonsh_config__ = {}
-    builtins.__xonsh_env__ = Env(default_env(config=config, login=login))
+    builtins.__xonsh_env__ = Env(default_env())
     builtins.__xonsh_help__ = helper
     builtins.__xonsh_superhelp__ = superhelper
     builtins.__xonsh_pathsearch__ = pathsearch
@@ -1170,8 +1174,6 @@ def load_builtins(execer=None, config=None, login=False, ctx=None):
     # Need this inline/lazy import here since we use locate_binary that
     # relies on __xonsh_env__ in default aliases
     builtins.default_aliases = builtins.aliases = Aliases(make_default_aliases())
-    if login:
-        builtins.aliases.update(load_foreign_aliases(issue_warning=False))
     builtins.__xonsh_history__ = None
     atexit.register(_lastflush)
     for sig in AT_EXIT_SIGNALS:

xonsh/codecache.py

@@ -101,7 +101,8 @@ def compile_code(filename, code, execer, glb, loc, mode):
             code += '\n'
         old_filename = execer.filename
         execer.filename = filename
-        ccode = execer.compile(code, glbs=glb, locs=loc, mode=mode)
+        ccode = execer.compile(code, glbs=glb, locs=loc, mode=mode,
+                               filename=filename)
     except Exception:
         raise
     finally:

xonsh/commands_cache.py

@@ -217,7 +217,7 @@ class CommandsCache(cabc.Mapping):
     def default_predictor(self, name, cmd0):
         if ON_POSIX:
             return self.default_predictor_readbin(name, cmd0,
-                                                  timeout=.1,
+                                                  timeout=0.1,
                                                   failure=predict_true)
         else:
             return predict_true
@@ -375,6 +375,7 @@ def default_threadable_predictors():
         'weechat': predict_help_ver,
         'xo': predict_help_ver,
         'xonsh': predict_shell,
+        'xon.sh': predict_shell,
         'zsh': predict_shell,
     }
     return predictors

xonsh/completers/path.py

@@ -125,7 +125,7 @@ def _quote_to_use(x):
         return single
 
 
-def _quote_paths(paths, start, end):
+def _quote_paths(paths, start, end, append_end=True):
     expand_path = builtins.__xonsh_expand_path__
     out = set()
     space = ' '
@@ -161,7 +161,8 @@ def _quote_paths(paths, start, end):
                 s += backslash
         if end in s:
             s = s.replace(end, ''.join('\\%s' % i for i in end))
-        out.add(start + s + end)
+        s = start + s + end if append_end else start + s
+        out.add(s)
     return out
 
 
@@ -194,8 +195,11 @@ def _splitpath_helper(path, sofar=()):
     folder, path = os.path.split(path)
     if path:
         sofar = sofar + (path, )
-    if (not folder or folder == xt.get_sep() or
-       (xp.ON_WINDOWS and os.path.splitdrive(path)[0])):
+    if not folder or folder == xt.get_sep():
+        return sofar[::-1]
+    elif xp.ON_WINDOWS and not path:
+        return os.path.splitdrive(folder)[:1] + sofar[::-1]
+    elif xp.ON_WINDOWS and os.path.splitdrive(path)[0]:
         return sofar[::-1]
     return _splitpath_helper(folder, sofar)
 
@@ -245,6 +249,7 @@ def complete_path(prefix, line, start, end, ctx, cdpath=True, filtfunc=None):
     # string stuff for automatic quoting
     path_str_start = ''
     path_str_end = ''
+    append_end = True
     p = _path_from_partial_string(line, end)
     lprefix = len(prefix)
     if p is not None:
@@ -252,6 +257,8 @@ def complete_path(prefix, line, start, end, ctx, cdpath=True, filtfunc=None):
         prefix = p[1]
         path_str_start = p[2]
         path_str_end = p[3]
+        if len(line) >= end + 1 and line[end] == path_str_end:
+            append_end = False
     tilde = '~'
     paths = set()
     env = builtins.__xonsh_env__
@@ -291,7 +298,8 @@ def complete_path(prefix, line, start, end, ctx, cdpath=True, filtfunc=None):
     paths = set(filter(filtfunc, paths))
     paths = _quote_paths({_normpath(s) for s in paths},
                          path_str_start,
-                         path_str_end)
+                         path_str_end,
+                         append_end)
     paths.update(filter(filtfunc, _dots(prefix)))
     paths.update(filter(filtfunc, _env(prefix)))
     return paths, lprefix

xonsh/completers/python.py

@@ -171,7 +171,10 @@ def python_signature_complete(prefix, line, end, ctx, filter_func):
     val, _ctx = _safe_eval(funcname, ctx)
     if val is None and _ctx is None:
         return set()
-    sig = inspect.signature(val)
+    try:
+        sig = inspect.signature(val)
+    except ValueError:
+        return set()
     args = {p + '=' for p in sig.parameters if filter_func(p, prefix)}
     return args
 

xonsh/environ.py

@@ -9,7 +9,6 @@ import textwrap
 import locale
 import builtins
 import warnings
-import traceback
 import contextlib
 import collections
 import collections.abc as cabc
@@ -19,7 +18,8 @@ from xonsh.lazyasd import LazyObject, lazyobject
 from xonsh.codecache import run_script_with_cache
 from xonsh.dirstack import _get_cwd
 from xonsh.events import events
-from xonsh.foreign_shells import load_foreign_envs
+from xonsh.foreign_shells import load_foreign_envs, load_foreign_aliases
+from xonsh.xontribs import update_context, prompt_xontrib_install
 from xonsh.platform import (
     BASH_COMPLETIONS_DEFAULT, DEFAULT_ENCODING, PATH_DEFAULT,
     ON_WINDOWS, ON_LINUX
@@ -39,6 +39,7 @@ from xonsh.tools import (
     is_logfile_opt, to_logfile_opt, logfile_opt_to_str, executables_in,
     is_nonstring_seq_of_strings, pathsep_to_upper_seq,
     seq_to_upper_pathsep, print_color, is_history_backend, to_itself,
+    swap_values,
 )
 import xonsh.prompt.base as prompt
 
@@ -162,7 +163,9 @@ def DEFAULT_ENSURERS():
     'BOTTOM_TOOLBAR': (is_string_or_callable, ensure_string, ensure_string),
     'SUBSEQUENCE_PATH_COMPLETION': (is_bool, to_bool, bool_to_str),
     'SUPPRESS_BRANCH_TIMEOUT_MESSAGE': (is_bool, to_bool, bool_to_str),
+    'UPDATE_COMPLETIONS_ON_KEYPRESS': (is_bool, to_bool, bool_to_str),
     'UPDATE_OS_ENVIRON': (is_bool, to_bool, bool_to_str),
+    'UPDATE_PROMPT_ON_KEYPRESS': (is_bool, to_bool, bool_to_str),
     'VC_BRANCH_TIMEOUT': (is_float, float, str),
     'VC_HG_SHOW_BRANCH': (is_bool, to_bool, bool_to_str),
     'VI_MODE': (is_bool, to_bool, bool_to_str),
@@ -188,7 +191,6 @@ def DEFAULT_ENSURERS():
     'XONSH_STORE_STDIN': (is_bool, to_bool, bool_to_str),
     'XONSH_TRACEBACK_LOGFILE': (is_logfile_opt, to_logfile_opt, logfile_opt_to_str),
     'XONSH_DATETIME_FORMAT': (is_string, ensure_string, ensure_string),
-    'UPDATE_PROMPT_ON_KEYPRESS': (is_bool, to_bool, bool_to_str),
     }
 
 
@@ -233,20 +235,19 @@ def xonshconfig(env):
     return xc
 
 
-def default_xonshrc():
+@default_value
+def default_xonshrc(env):
     """Creates a new instance of the default xonshrc tuple."""
     if ON_WINDOWS:
-        dxrc = (os.path.join(os.environ['ALLUSERSPROFILE'],
+        dxrc = (xonshconfig(env),
+                os.path.join(os.environ['ALLUSERSPROFILE'],
                              'xonsh', 'xonshrc'),
                 os.path.expanduser('~/.xonshrc'))
     else:
-        dxrc = ('/etc/xonshrc', os.path.expanduser('~/.xonshrc'))
+        dxrc = (xonshconfig(env), '/etc/xonshrc', os.path.expanduser('~/.xonshrc'))
     return dxrc
 
 
-DEFAULT_XONSHRC = LazyObject(default_xonshrc, globals(), 'DEFAULT_XONSHRC')
-
-
 # Default values should generally be immutable, that way if a user wants
 # to set them they have to do a copy and write them to the environment.
 # try to keep this sorted.
@@ -305,7 +306,9 @@ def DEFAULT_VALUES():
         'SUGGEST_MAX_NUM': 5,
         'SUGGEST_THRESHOLD': 3,
         'TITLE': DEFAULT_TITLE,
+        'UPDATE_COMPLETIONS_ON_KEYPRESS': False,
         'UPDATE_OS_ENVIRON': False,
+        'UPDATE_PROMPT_ON_KEYPRESS': False,
         'VC_BRANCH_TIMEOUT': 0.2 if ON_WINDOWS else 0.1,
         'VC_HG_SHOW_BRANCH': True,
         'VI_MODE': False,
@@ -314,7 +317,7 @@ def DEFAULT_VALUES():
         'XDG_DATA_HOME': os.path.expanduser(os.path.join('~', '.local',
                                                          'share')),
         'XONSHCONFIG': xonshconfig,
-        'XONSHRC': default_xonshrc(),
+        'XONSHRC': default_xonshrc,
         'XONSH_AUTOPAIR': False,
         'XONSH_CACHE_SCRIPTS': True,
         'XONSH_CACHE_EVERYTHING': False,
@@ -336,7 +339,6 @@ def DEFAULT_VALUES():
         'XONSH_STORE_STDOUT': False,
         'XONSH_TRACEBACK_LOGFILE': None,
         'XONSH_DATETIME_FORMAT': '%Y-%m-%d %H:%M',
-        'UPDATE_PROMPT_ON_KEYPRESS': False,
     }
     if hasattr(locale, 'LC_MESSAGES'):
         dv['LC_MESSAGES'] = locale.setlocale(locale.LC_MESSAGES)
@@ -424,7 +426,7 @@ def DEFAULT_DOCS():
         "- If ``$COMPLETIONS_DISPLAY`` is ``multi`` or ``true``, display completions\n"
         "  in multiple columns while typing.\n\n"
         'These option values are not case- or type-sensitive, so e.g.'
-        "writing ``$COMPLETIONS_DISPLAY = None``"
+        "writing ``$COMPLETIONS_DISPLAY = None`` "
         "and ``$COMPLETIONS_DISPLAY = 'none'`` are equivalent. Only usable with "
         "``$SHELL_TYPE=prompt_toolkit``"),
     'COMPLETIONS_CONFIRM': VarDocs(
@@ -591,10 +593,19 @@ def DEFAULT_DOCS():
         "in the same manner as ``$PROMPT``, see 'Customizing the Prompt' "
         'http://xon.sh/tutorial.html#customizing-the-prompt.',
         default='``xonsh.environ.DEFAULT_TITLE``'),
+    'UPDATE_COMPLETIONS_ON_KEYPRESS': VarDocs(
+        'Completions display is evaluated and presented whenever a key is '
+        'pressed. This avoids the need to press TAB, except to cycle through '
+        'the possibilities. This currently only affects the prompt-toolkit shell.'
+        ),
     'UPDATE_OS_ENVIRON': VarDocs(
         "If True ``os.environ`` will always be updated "
         "when the xonsh environment changes. The environment can be reset to "
         "the default value by calling ``__xonsh_env__.undo_replace_env()``"),
+    'UPDATE_PROMPT_ON_KEYPRESS': VarDocs(
+        'Disables caching the prompt between commands, '
+        'so that it would be reevaluated on each keypress. '
+        'Disabled by default because of the incurred performance penalty.'),
     'VC_BRANCH_TIMEOUT': VarDocs(
         'The timeout (in seconds) for version control '
         'branch computations. This is a timeout per subprocess call, so the '
@@ -740,10 +751,6 @@ def DEFAULT_DOCS():
     'XONSH_DATETIME_FORMAT': VarDocs(
         'The format that is used for ``datetime.strptime()`` in various places'
         'i.e the history timestamp option'),
-    'UPDATE_PROMPT_ON_KEYPRESS': VarDocs(
-        'Disables caching the prompt between commands, '
-        'so that it would be reevaluated on each keypress. '
-        'Disabled by default because of the incurred performance penalty.'),
     }
 
 
@@ -1070,36 +1077,26 @@ def load_static_config(ctx, config=None):
     return conf
 
 
-def xonshrc_context(rcfiles=None, execer=None, initial=None):
-    """Attempts to read in xonshrc file, and return the contents."""
-    loaded = builtins.__xonsh_env__['LOADED_RC_FILES'] = []
-    if initial is None:
-        env = {}
-    else:
-        env = initial
-    if rcfiles is None or execer is None:
+def xonshrc_context(rcfiles=None, execer=None, ctx=None, env=None, login=True):
+    """Attempts to read in all xonshrc files and return the context."""
+    loaded = env['LOADED_RC_FILES'] = []
+    ctx = {} if ctx is None else ctx
+    if rcfiles is None:
         return env
     env['XONSHRC'] = tuple(rcfiles)
     for rcfile in rcfiles:
         if not os.path.isfile(rcfile):
             loaded.append(False)
             continue
-        try:
-            run_script_with_cache(rcfile, execer, env)
-            loaded.append(True)
-        except SyntaxError as err:
-            loaded.append(False)
-            exc = traceback.format_exc()
-            msg = '{0}\nsyntax error in xonsh run control file {1!r}: {2!s}'
-            warnings.warn(msg.format(exc, rcfile, err), RuntimeWarning)
-            continue
-        except Exception as err:
-            loaded.append(False)
-            exc = traceback.format_exc()
-            msg = '{0}\nerror running xonsh run control file {1!r}: {2!s}'
-            warnings.warn(msg.format(exc, rcfile, err), RuntimeWarning)
-            continue
-    return env
+        _, ext = os.path.splitext(rcfile)
+        if ext == '.json':
+            status = static_config_run_control(rcfile, ctx, env, execer=execer,
+                                               login=login)
+        else:
+            status = xonsh_script_run_control(rcfile, ctx, env, execer=execer,
+                                              login=login)
+        loaded.append(status)
+    return ctx
 
 
 def windows_foreign_env_fixes(ctx):
@@ -1125,7 +1122,62 @@ def foreign_env_fixes(ctx):
         del ctx['PROMPT']
 
 
-def default_env(env=None, config=None, login=True):
+def static_config_run_control(filename, ctx, env, execer=None, login=True):
+    """Loads a static config file and applies it as a run control."""
+    if not login:
+        return
+    conf = load_static_config(env, config=filename)
+    # load foreign shells
+    foreign_env = load_foreign_envs(shells=conf.get('foreign_shells', ()),
+                                    issue_warning=False)
+    if ON_WINDOWS:
+        windows_foreign_env_fixes(foreign_env)
+    foreign_env_fixes(foreign_env)
+    env.update(foreign_env)
+    aliases = builtins.aliases
+    foreign_aliases = load_foreign_aliases(config=filename, issue_warning=True)
+    for k, v in foreign_aliases.items():
+        if k in aliases:
+            msg = ('Skipping application of {0!r} alias from foreign shell '
+                   '(loaded from {1!r}) since it shares a name with an '
+                   'existing xonsh alias.')
+            print(msg.format(k, filename))
+        else:
+            aliases[k] = v
+    # load xontribs
+    names = conf.get('xontribs', ())
+    for name in names:
+        update_context(name, ctx=ctx)
+    if getattr(update_context, 'bad_imports', None):
+        prompt_xontrib_install(update_context.bad_imports)
+        del update_context.bad_imports
+    # Do static config environment last, to allow user to override any of
+    # our environment choices
+    env.update(conf.get('env', ()))
+    return True
+
+
+def xonsh_script_run_control(filename, ctx, env, execer=None, login=True):
+    """Loads a xonsh file and applies it as a run control."""
+    if execer is None:
+        return False
+    updates = {'__file__': filename, '__name__': os.path.abspath(filename)}
+    try:
+        with swap_values(ctx, updates):
+            run_script_with_cache(filename, execer, ctx)
+        loaded = True
+    except SyntaxError as err:
+        msg = 'syntax error in xonsh run control file {0!r}: {1!s}'
+        print_exception(msg.format(filename, err))
+        loaded = False
+    except Exception as err:
+        msg = 'error running xonsh run control file {0!r}: {1!s}'
+        print_exception(msg.format(filename, err))
+        loaded = False
+    return loaded
+
+
+def default_env(env=None):
     """Constructs a default xonsh environment."""
     # in order of increasing precedence
     ctx = dict(BASE_ENV)
@@ -1136,17 +1188,6 @@ def default_env(env=None, config=None, login=True):
         del ctx['PROMPT']
     except KeyError:
         pass
-    if login:
-        conf = load_static_config(ctx, config=config)
-        foreign_env = load_foreign_envs(shells=conf.get('foreign_shells', ()),
-                                        issue_warning=False)
-        if ON_WINDOWS:
-            windows_foreign_env_fixes(foreign_env)
-        foreign_env_fixes(foreign_env)
-        ctx.update(foreign_env)
-        # Do static config environment last, to allow user to override any of
-        # our environment choices
-        ctx.update(conf.get('env', ()))
     # finalize env
     if env is not None:
         ctx.update(env)

xonsh/execer.py

@@ -8,7 +8,8 @@ import collections.abc as cabc
 
 from xonsh.ast import CtxAwareTransformer
 from xonsh.parser import Parser
-from xonsh.tools import subproc_toks, find_next_break
+from xonsh.tools import (subproc_toks, find_next_break, get_logical_line,
+                         replace_logical_line)
 from xonsh.built_ins import load_builtins, unload_builtins
 
 
@@ -16,7 +17,7 @@ class Execer(object):
     """Executes xonsh code in a context."""
 
     def __init__(self, filename='<xonsh-code>', debug_level=0, parser_args=None,
-                 unload=True, config=None, login=True, xonsh_ctx=None):
+                 unload=True, xonsh_ctx=None, scriptcache=True, cacheall=False):
         """Parameters
         ----------
         filename : str, optional
@@ -27,18 +28,24 @@ class Execer(object):
             Arguments to pass down to the parser.
         unload : bool, optional
             Whether or not to unload xonsh builtins upon deletion.
-        config : str, optional
-            Path to configuration file.
         xonsh_ctx : dict or None, optional
             Xonsh xontext to load as builtins.__xonsh_ctx__
+        scriptcache : bool, optional
+            Whether or not to use a precompiled bytecode cache when execing
+            code, default: True.
+        cacheall : bool, optional
+            Whether or not to cache all xonsh code, and not just files. If this
+            is set to true, it will cache command line input too, default: False.
         """
         parser_args = parser_args or {}
         self.parser = Parser(**parser_args)
         self.filename = filename
         self.debug_level = debug_level
         self.unload = unload
+        self.scriptcache = scriptcache
+        self.cacheall = cacheall
         self.ctxtransformer = CtxAwareTransformer(self.parser)
-        load_builtins(execer=self, config=config, login=login, ctx=xonsh_ctx)
+        load_builtins(execer=self, ctx=xonsh_ctx)
 
     def __del__(self):
         if self.unload:
@@ -172,7 +179,7 @@ class Execer(object):
                 last_error_line = e.loc.lineno
                 idx = last_error_line - 1
                 lines = input.splitlines()
-                line = lines[idx]
+                line, nlogical = get_logical_line(lines, idx)
                 if input.endswith('\n'):
                     lines.append('')
                 if len(line.strip()) == 0:
@@ -214,6 +221,7 @@ class Execer(object):
                     # anything
                     raise original_error
                 else:
+                    # print some debugging info
                     if self.debug_level > 1:
                         msg = ('{0}:{1}:{2}{3} - {4}\n'
                                '{0}:{1}:{2}{3} + {5}')
@@ -221,7 +229,8 @@ class Execer(object):
                         msg = msg.format(self.filename, last_error_line,
                                          last_error_col, mstr, line, sbpline)
                         print(msg, file=sys.stderr)
-                    lines[idx] = sbpline
+                    # replace the line
+                    replace_logical_line(lines, sbpline, idx, nlogical)
                 last_error_col += 3
                 input = '\n'.join(lines)
         return tree, input

xonsh/imphooks.py

@@ -3,13 +3,16 @@
 
 This module registers the hooks it defines when it is imported.
 """
-import builtins
 import os
 import sys
 import types
+import builtins
+import contextlib
+import importlib
 from importlib.machinery import ModuleSpec
-from importlib.abc import MetaPathFinder, SourceLoader
+from importlib.abc import MetaPathFinder, SourceLoader, Loader
 
+from xonsh.events import events
 from xonsh.execer import Execer
 from xonsh.platform import scandir
 
@@ -93,16 +96,170 @@ class XonshImportHook(MetaPathFinder, SourceLoader):
         return code
 
 
-def install_hook():
-    """
-    Install Xonsh import hook in `sys.metapath` in order for `.xsh` files to be
-    importable.
+#
+# Import events
+#
+events.doc('on_import_pre_find_spec', """
+on_import_pre_find_spec(fullname: str, path: str, target: module or None) -> None
 
-    Can safely be called many times, will be no-op if a xonsh import hook is
+Fires before any import find_spec() calls have been executed. The parameters
+here are the same as importlib.abc.MetaPathFinder.find_spec(). Namely,
+
+``fullname``: The full name of the module to import.
+``path``: None if a top-level import, otherwise the ``__path__`` of the parent
+          package.
+``target``: Target module used to make a better guess about the package spec.
+""")
+
+events.doc('on_import_post_find_spec', """
+on_import_post_find_spec(spec, fullname, path, target) -> None
+
+Fires after all import find_spec() calls have been executed. The parameters
+here the spec and the arguments importlib.abc.MetaPathFinder.find_spec(). Namely,
+
+``spec``: A ModuleSpec object if the spec was found, or None if it was not.
+``fullname``: The full name of the module to import.
+``path``: None if a top-level import, otherwise the ``__path__`` of the parent
+          package.
+``target``: Target module used to make a better guess about the package spec.
+""")
+
+events.doc('on_import_pre_create_module', """
+on_import_pre_create_module(spec: ModuleSpec) -> None
+
+Fires right before a module is created by its loader. The only parameter
+is the spec object. See importlib for more details.
+""")
+
+events.doc('on_import_post_create_module', """
+on_import_post_create_module(module: Module, spec: ModuleSpec) -> None
+
+Fires after a module is created by its loader but before the loader returns it.
+The parameters here are the module object itself and the spec object.
+See importlib for more details.
+""")
+
+events.doc('on_import_pre_exec_module', """
+on_import_pre_exec_module(module: Module) -> None
+
+Fires right before a module is executed by its loader. The only parameter
+is the module itself. See importlib for more details.
+""")
+
+events.doc('on_import_post_exec_module', """
+on_import_post_create_module(module: Module) -> None
+
+Fires after a module is executed by its loader but before the loader returns it.
+The only parameter is the module itself. See importlib for more details.
+""")
+
+
+def _should_dispatch_xonsh_import_event_loader():
+    """Figures out if we should dispatch to a load event"""
+    return (len(events.on_import_pre_create_module) > 0 or
+            len(events.on_import_post_create_module) > 0 or
+            len(events.on_import_pre_exec_module) > 0 or
+            len(events.on_import_post_exec_module) > 0)
+
+
+class XonshImportEventHook(MetaPathFinder):
+    """Implements the import hook for firing xonsh events on import."""
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._fullname_stack = []
+
+    @contextlib.contextmanager
+    def append_stack(self, fullname):
+        """A context manager for appending and then removing a name from the
+        fullname stack.
+        """
+        self._fullname_stack.append(fullname)
+        yield
+        del self._fullname_stack[-1]
+
+    #
+    # MetaPathFinder methods
+    #
+    def find_spec(self, fullname, path, target=None):
+        """Finds the spec for a xonsh module if it exists."""
+        if fullname in reversed(self._fullname_stack):
+            # don't execute if we are already in the stack.
+            return None
+        npre = len(events.on_import_pre_find_spec)
+        npost = len(events.on_import_post_find_spec)
+        dispatch_load = _should_dispatch_xonsh_import_event_loader()
+        if npre > 0:
+            events.on_import_pre_find_spec.fire(fullname=fullname, path=path,
+                                                target=target)
+        elif npost == 0 and not dispatch_load:
+            # no events to fire, proceed normally and prevent recursion
+            return None
+        # now find the spec
+        with self.append_stack(fullname):
+            spec = importlib.util.find_spec(fullname)
+        # fire post event
+        if npost > 0:
+            events.on_import_post_find_spec.fire(spec=spec, fullname=fullname,
+                                                 path=path, target=target)
+        if dispatch_load and spec is not None and hasattr(spec.loader, 'create_module'):
+            spec.loader = XonshImportEventLoader(spec.loader)
+        return spec
+
+
+class XonshImportEventLoader(Loader):
+    """A class that dispatches loader calls to another loader and fires relevent
+    xonsh events.
+    """
+
+    def __init__(self, loader):
+        self.loader = loader
+
+    #
+    # Loader methods
+    #
+    def create_module(self, spec):
+        """Creates and returns the module object."""
+        events.on_import_pre_create_module.fire(spec=spec)
+        mod = self.loader.create_module(spec)
+        events.on_import_post_create_module.fire(module=mod, spec=spec)
+        return mod
+
+    def exec_module(self, module):
+        """Executes the module in its own namespace."""
+        events.on_import_pre_exec_module.fire(module=module)
+        rtn = self.loader.exec_module(module)
+        events.on_import_post_exec_module.fire(module=module)
+        return rtn
+
+    def load_module(self, fullname):
+        """Legacy module loading, provided for backwards compatability."""
+        return self.loader.load_module(fullname)
+
+    def module_repr(self, module):
+        """Legacy module repr, provided for backwards compatability."""
+        return self.loader.module_repr(module)
+
+
+def install_import_hooks():
+    """
+    Install Xonsh import hooks in ``sys.meta_path`` in order for ``.xsh`` files
+    to be importable and import events to be fired.
+
+    Can safely be called many times, will be no-op if xonsh import hooks are
     already present.
     """
+    found_imp = found_event = False
     for hook in sys.meta_path:
         if isinstance(hook, XonshImportHook):
-            break
-    else:
+            found_imp = True
+        elif isinstance(hook, XonshImportEventHook):
+            found_event = True
+    if not found_imp:
         sys.meta_path.append(XonshImportHook())
+    if not found_event:
+        sys.meta_path.insert(0, XonshImportEventHook())
+
+
+# alias to deprecated name
+install_hook = install_import_hooks

xonsh/jobs.py

@@ -6,12 +6,12 @@ import time
 import ctypes
 import signal
 import builtins
-import functools
 import subprocess
 import collections
 
 from xonsh.lazyasd import LazyObject
-from xonsh.platform import ON_DARWIN, ON_WINDOWS, ON_CYGWIN, LIBC
+from xonsh.platform import FD_STDERR, ON_DARWIN, ON_WINDOWS, ON_CYGWIN, LIBC
+from xonsh.tools import unthreadable
 
 
 tasks = LazyObject(collections.deque, globals(), 'tasks')
@@ -28,7 +28,10 @@ if ON_DARWIN:
         for pid in job['pids']:
             if pid is None:  # the pid of an aliased proc is None
                 continue
-            os.kill(pid, signal)
+            try:
+                os.kill(pid, signal)
+            except ProcessLookupError:
+                pass
 elif ON_WINDOWS:
     pass
 elif ON_CYGWIN:
@@ -67,7 +70,7 @@ if ON_WINDOWS:
     def ignore_sigtstp():
         pass
 
-    def _set_pgrp(info):
+    def give_terminal_to(pgid):
         pass
 
     def wait_for_active_job(last_task=None, backgrounded=False):
@@ -101,64 +104,49 @@ else:
     def ignore_sigtstp():
         signal.signal(signal.SIGTSTP, signal.SIG_IGN)
 
-    def _set_pgrp(info):
-        pid = info['pids'][0]
-        if pid is None:
-            # occurs if first process is an alias
-            info['pgrp'] = None
-            return
-        try:
-            info['pgrp'] = os.getpgid(pid)
-        except ProcessLookupError:
-            info['pgrp'] = None
-
     _shell_pgrp = os.getpgrp()
 
     _block_when_giving = LazyObject(lambda: (signal.SIGTTOU, signal.SIGTTIN,
                                              signal.SIGTSTP, signal.SIGCHLD),
                                     globals(), '_block_when_giving')
 
-    # check for shell tty
-    @functools.lru_cache(1)
-    def _shell_tty():
-        try:
-            _st = sys.stderr.fileno()
-            if os.tcgetpgrp(_st) != os.getpgid(os.getpid()):
-                # we don't own it
-                _st = None
-        except OSError:
-            _st = None
-        return _st
-
-    # _give_terminal_to is a simplified version of:
+    # give_terminal_to is a simplified version of:
     #    give_terminal_to from bash 4.3 source, jobs.c, line 4030
     # this will give the terminal to the process group pgid
     if ON_CYGWIN:
         # on cygwin, signal.pthread_sigmask does not exist in Python, even
         # though pthread_sigmask is defined in the kernel.  thus, we use
         # ctypes to mimic the calls in the "normal" version below.
-        def _give_terminal_to(pgid):
-            st = _shell_tty()
-            if st is not None and os.isatty(st):
-                omask = ctypes.c_ulong()
-                mask = ctypes.c_ulong()
-                LIBC.sigemptyset(ctypes.byref(mask))
-                for i in _block_when_giving:
-                    LIBC.sigaddset(ctypes.byref(mask), ctypes.c_int(i))
-                LIBC.sigemptyset(ctypes.byref(omask))
-                LIBC.sigprocmask(ctypes.c_int(signal.SIG_BLOCK),
-                                 ctypes.byref(mask),
-                                 ctypes.byref(omask))
-                LIBC.tcsetpgrp(ctypes.c_int(st), ctypes.c_int(pgid))
-                LIBC.sigprocmask(ctypes.c_int(signal.SIG_SETMASK),
-                                 ctypes.byref(omask), None)
+        def give_terminal_to(pgid):
+            omask = ctypes.c_ulong()
+            mask = ctypes.c_ulong()
+            LIBC.sigemptyset(ctypes.byref(mask))
+            for i in _block_when_giving:
+                LIBC.sigaddset(ctypes.byref(mask), ctypes.c_int(i))
+            LIBC.sigemptyset(ctypes.byref(omask))
+            LIBC.sigprocmask(ctypes.c_int(signal.SIG_BLOCK),
+                             ctypes.byref(mask),
+                             ctypes.byref(omask))
+            LIBC.tcsetpgrp(ctypes.c_int(FD_STDERR), ctypes.c_int(pgid))
+            LIBC.sigprocmask(ctypes.c_int(signal.SIG_SETMASK),
+                             ctypes.byref(omask), None)
+            return True
     else:
-        def _give_terminal_to(pgid):
-            st = _shell_tty()
-            if st is not None and os.isatty(st):
-                oldmask = signal.pthread_sigmask(signal.SIG_BLOCK,
-                                                 _block_when_giving)
-                os.tcsetpgrp(st, pgid)
+        def give_terminal_to(pgid):
+            oldmask = signal.pthread_sigmask(signal.SIG_BLOCK,
+                                             _block_when_giving)
+            try:
+                os.tcsetpgrp(FD_STDERR, pgid)
+                return True
+            except OSError as e:
+                if e.errno == 22:  # [Errno 22] Invalid argument
+                    # there are cases that all the processes of pgid have
+                    # finished, then we don't need to do anything here, see
+                    # issue #2220
+                    return False
+                else:
+                    raise
+            finally:
                 signal.pthread_sigmask(signal.SIG_SETMASK, oldmask)
 
     def wait_for_active_job(last_task=None, backgrounded=False):
@@ -170,22 +158,14 @@ else:
         active_task = get_next_task()
         # Return when there are no foreground active task
         if active_task is None:
-            _give_terminal_to(_shell_pgrp)  # give terminal back to the shell
-            if backgrounded and hasattr(builtins, '__xonsh_shell__'):
-                # restoring sanity could probably be called whenever we return
-                # control to the shell. But it only seems to matter after a
-                # ^Z event. This *has* to be called after we give the terminal
-                # back to the shell.
-                builtins.__xonsh_shell__.shell.restore_tty_sanity()
             return last_task
-        pgrp = active_task.get('pgrp', None)
         obj = active_task['obj']
         backgrounded = False
-        # give the terminal over to the fg process
-        if pgrp is not None:
-            _give_terminal_to(pgrp)
-        _continue(active_task)
-        _, wcode = os.waitpid(obj.pid, os.WUNTRACED)
+        try:
+            _, wcode = os.waitpid(obj.pid, os.WUNTRACED)
+        except ChildProcessError:  # No child processes
+            return wait_for_active_job(last_task=active_task,
+                                       backgrounded=backgrounded)
         if os.WIFSTOPPED(wcode):
             print('^Z')
             active_task['status'] = "stopped"
@@ -262,7 +242,6 @@ def add_job(info):
     num = get_next_job_number()
     info['started'] = time.time()
     info['status'] = "running"
-    _set_pgrp(info)
     tasks.appendleft(num)
     builtins.__xonsh_all_jobs__[num] = info
     if info['bg']:
@@ -307,9 +286,9 @@ def clean_jobs():
                     # newline
                     print()
                 print('xonsh: {}'.format(msg), file=sys.stderr)
-                print('-'*5, file=sys.stderr)
+                print('-' * 5, file=sys.stderr)
                 jobs([], stdout=sys.stderr)
-                print('-'*5, file=sys.stderr)
+                print('-' * 5, file=sys.stderr)
                 print('Type "exit" or press "ctrl-d" again to force quit.',
                       file=sys.stderr)
                 jobs_clean = False
@@ -341,6 +320,7 @@ def jobs(args, stdin=None, stdout=sys.stdout, stderr=None):
     return None, None
 
 
+@unthreadable
 def fg(args, stdin=None):
     """
     xonsh command: fg
@@ -354,32 +334,33 @@ def fg(args, stdin=None):
         return '', 'Cannot bring nonexistent job to foreground.\n'
 
     if len(args) == 0:
-        act = tasks[0]  # take the last manipulated task by default
+        tid = tasks[0]  # take the last manipulated task by default
     elif len(args) == 1:
         try:
             if args[0] == '+':  # take the last manipulated task
-                act = tasks[0]
+                tid = tasks[0]
             elif args[0] == '-':  # take the second to last manipulated task
-                act = tasks[1]
+                tid = tasks[1]
             else:
-                act = int(args[0])
+                tid = int(args[0])
         except (ValueError, IndexError):
             return '', 'Invalid job: {}\n'.format(args[0])
 
-        if act not in builtins.__xonsh_all_jobs__:
+        if tid not in builtins.__xonsh_all_jobs__:
             return '', 'Invalid job: {}\n'.format(args[0])
     else:
         return '', 'fg expects 0 or 1 arguments, not {}\n'.format(len(args))
 
     # Put this one on top of the queue
-    tasks.remove(act)
-    tasks.appendleft(act)
+    tasks.remove(tid)
+    tasks.appendleft(tid)
 
-    job = get_task(act)
+    job = get_task(tid)
     job['bg'] = False
     job['status'] = "running"
-    print_one_job(act)
-    wait_for_active_job()
+    print_one_job(tid)
+    pipeline = job['pipeline']
+    pipeline.resume(job)
 
 
 def bg(args, stdin=None):

xonsh/lexer.py

@@ -334,6 +334,33 @@ class Lexer(object):
             yield t
             t = self.token()
 
+    def split(self, s):
+        """Splits a string into a list of strings which are whitepace-separated
+        tokens.
+        """
+        vals = []
+        self.input(s)
+        l = c = -1
+        ws = 'WS'
+        nl = '\n'
+        for t in self:
+            if t.type == ws:
+                continue
+            elif l < t.lineno:
+                vals.append(t.value)
+            elif len(vals) > 0 and c == t.lexpos:
+                vals[-1] = vals[-1] + t.value
+            else:
+                vals.append(t.value)
+            nnl = t.value.count(nl)
+            if nnl == 0:
+                l = t.lineno
+                c = t.lexpos + len(t.value)
+            else:
+                l = t.lineno + nnl
+                c = len(t.value.rpartition(nl)[-1])
+        return vals
+
     #
     # All the tokens recognized by the lexer
     #

xonsh/main.py

@@ -6,21 +6,25 @@ import enum
 import argparse
 import builtins
 import contextlib
+import signal
 import traceback
 
 from xonsh import __version__
+from xonsh.timings import setup_timings
 from xonsh.lazyasd import lazyobject
 from xonsh.shell import Shell
 from xonsh.pretty import pretty
+from xonsh.execer import Execer
 from xonsh.proc import HiddenCommandPipeline
 from xonsh.jobs import ignore_sigtstp
-from xonsh.tools import setup_win_unicode_console, print_color
+from xonsh.tools import setup_win_unicode_console, print_color, to_bool_or_int
 from xonsh.platform import HAS_PYGMENTS, ON_WINDOWS
 from xonsh.codecache import run_script_with_cache, run_code_with_cache
 from xonsh.xonfig import xonfig_main
 from xonsh.lazyimps import pygments, pyghooks
-from xonsh.imphooks import install_hook
+from xonsh.imphooks import install_import_hooks
 from xonsh.events import events
+from xonsh.environ import xonshrc_context
 
 
 events.transmogrify('on_post_init', 'LoadEvent')
@@ -124,10 +128,18 @@ def parser():
                    action='store_true',
                    default=False)
     p.add_argument('--config-path',
-                   help='specify a custom static configuration file',
+                   help='DEPRECATED: static configuration files may now be used '
+                        'in the XONSHRC file list, see the --rc option.',
                    dest='config_path',
                    default=None,
                    type=path_argument)
+    p.add_argument('--rc',
+                   help="The xonshrc files to load, these may be either xonsh "
+                        "files or JSON-based static configuration files.",
+                   dest='rc',
+                   nargs='+',
+                   type=path_argument,
+                   default=None)
     p.add_argument('--no-rc',
                    help="Do not load the .xonshrc files",
                    dest='norc',
@@ -157,6 +169,13 @@ def parser():
                    dest='shell_type',
                    choices=('readline', 'prompt_toolkit', 'best', 'random'),
                    default=None)
+    p.add_argument('--timings',
+                   help='Prints timing infomation before the prompt is shown. '
+                        'This is usefull to track down perfomance issues '
+                        'and investigate startup times.',
+                   dest='timings',
+                   action='store_true',
+                   default=None)
     p.add_argument('file',
                    metavar='script-file',
                    help='If present, execute the script in script-file'
@@ -199,10 +218,38 @@ class XonshMode(enum.Enum):
     interactive = 3
 
 
+def start_services(shell_kwargs):
+    """Starts up the essential services in the proper order.
+    This returns the envrionment instance as a convenience.
+    """
+    install_import_hooks()
+    # create execer, which loads builtins
+    ctx = shell_kwargs.get('ctx', {})
+    debug = to_bool_or_int(os.getenv('XONSH_DEBUG', '0'))
+    events.on_timingprobe.fire(name='pre_execer_init')
+    execer = Execer(xonsh_ctx=ctx, debug_level=debug,
+                    scriptcache=shell_kwargs.get('scriptcache', True),
+                    cacheall=shell_kwargs.get('cacheall', False))
+    events.on_timingprobe.fire(name='post_execer_init')
+    # load rc files
+    login = shell_kwargs.get('login', True)
+    env = builtins.__xonsh_env__
+    rc = shell_kwargs.get('rc', None)
+    rc = env.get('XONSHRC') if rc is None else rc
+    events.on_pre_rc.fire()
+    xonshrc_context(rcfiles=rc, execer=execer, ctx=ctx, env=env, login=login)
+    events.on_post_rc.fire()
+    # create shell
+    builtins.__xonsh_shell__ = Shell(execer=execer, **shell_kwargs)
+    ctx['__name__'] = '__main__'
+    return env
+
+
 def premain(argv=None):
     """Setup for main xonsh entry point, returns parsed arguments."""
     if argv is None:
         argv = sys.argv[1:]
+    setup_timings()
     setproctitle = get_setproctitle()
     if setproctitle is not None:
         setproctitle(' '.join(['xonsh'] + argv))
@@ -223,8 +270,6 @@ def premain(argv=None):
                     'ctx': builtins.__xonsh_ctx__}
     if args.login:
         shell_kwargs['login'] = True
-    if args.config_path is not None:
-        shell_kwargs['config'] = args.config_path
     if args.norc:
         shell_kwargs['rc'] = ()
     setattr(sys, 'displayhook', _pprint_displayhook)
@@ -241,9 +286,7 @@ def premain(argv=None):
         args.mode = XonshMode.interactive
         shell_kwargs['completer'] = True
         shell_kwargs['login'] = True
-    install_hook()
-    builtins.__xonsh_shell__ = Shell(**shell_kwargs)
-    env = builtins.__xonsh_env__
+    env = start_services(shell_kwargs)
     env['XONSH_LOGIN'] = shell_kwargs['login']
     if args.defines is not None:
         env.update([x.split('=', 1) for x in args.defines])
@@ -298,6 +341,12 @@ def main(argv=None):
 
 def main_xonsh(args):
     """Main entry point for xonsh cli."""
+    if not ON_WINDOWS:
+        def func_sig_ttin_ttou(n, f):
+            pass
+        signal.signal(signal.SIGTTIN, func_sig_ttin_ttou)
+        signal.signal(signal.SIGTTOU, func_sig_ttin_ttou)
+
     events.on_post_init.fire()
     env = builtins.__xonsh_env__
     shell = builtins.__xonsh_shell__
@@ -327,6 +376,7 @@ def main_xonsh(args):
                 sys.argv = [args.file] + args.args
                 env['ARGS'] = sys.argv[:]  # $ARGS is not sys.argv
                 env['XONSH_SOURCE'] = path
+                shell.ctx.update({'__file__': args.file, '__name__': '__main__'})
                 run_script_with_cache(args.file, shell.execer, glb=shell.ctx,
                                       loc=None, mode='exec')
             else:

xonsh/platform.py

@@ -14,6 +14,11 @@ import subprocess
 import importlib.util
 
 from xonsh.lazyasd import LazyBool, lazyobject, lazybool
+# do not import any xonsh-modules here to avoid circular dependencies
+
+FD_STDIN = 0
+FD_STDOUT = 1
+FD_STDERR = 2
 
 
 @lazyobject
@@ -27,9 +32,6 @@ def distro():
     return d
 
 
-# do not import any xonsh-modules here to avoid circular dependencies
-
-
 #
 # OS
 #
@@ -176,6 +178,7 @@ def pathbasename(p):
     """
     return pathsplit(p)[-1]
 
+
 # termios tc(get|set)attr indexes.
 IFLAG = 0
 OFLAG = 1
@@ -296,22 +299,28 @@ def windows_bash_command():
     # Check that bash is on path otherwise try the default directory
     # used by Git for windows
     wbc = 'bash'
-    bash_on_path = builtins.__xonsh_commands_cache__.lazy_locate_binary('bash',
-                                                                        ignore_alias=True)
+    cmd_cache = builtins.__xonsh_commands_cache__
+    bash_on_path = cmd_cache.lazy_locate_binary('bash', ignore_alias=True)
     if bash_on_path:
-        # Check if Bash is from the "Windows Subsystem for Linux" (WSL)
-        # which can't be used by xonsh foreign-shell/completer
-        out = subprocess.check_output([bash_on_path, '--version'],
-                                      stderr=subprocess.PIPE,
-                                      universal_newlines=True)
-        if 'pc-linux-gnu' in out.splitlines()[0]:
+        try:
+            out = subprocess.check_output([bash_on_path, '--version'],
+                                          stderr=subprocess.PIPE,
+                                          universal_newlines=True)
+        except subprocess.CalledProcessError:
+            bash_works = False
+        else:
+            # Check if Bash is from the "Windows Subsystem for Linux" (WSL)
+            # which can't be used by xonsh foreign-shell/completer
+            bash_works = 'pc-linux-gnu' not in out.splitlines()[0]
+
+        if bash_works:
+            wbc = bash_on_path
+        else:
             gfwp = git_for_windows_path()
             if gfwp:
                 bashcmd = os.path.join(gfwp, 'bin\\bash.exe')
                 if os.path.isfile(bashcmd):
                     wbc = bashcmd
-        else:
-            wbc = bash_on_path
     return wbc
 
 #

xonsh/proc.py

@@ -29,8 +29,13 @@ from xonsh.tools import (redirect_stdout, redirect_stderr, print_exception,
                          XonshCalledProcessError, findfirst, on_main_thread,
                          XonshError, format_std_prepost)
 from xonsh.lazyasd import lazyobject, LazyObject
-from xonsh.jobs import wait_for_active_job
+from xonsh.jobs import wait_for_active_job, give_terminal_to, _continue
 from xonsh.lazyimps import fcntl, termios, _winapi, msvcrt, winutils
+# these decorators are imported for users back-compatible
+from xonsh.tools import unthreadable, uncapturable  # NOQA
+
+# foreground has be deprecated
+foreground = unthreadable
 
 
 @lazyobject
@@ -327,11 +332,11 @@ def populate_console(reader, fd, buffer, chunksize, queue, expandsize=None):
     # I believe that there is a bug in PTK that if we reset the
     # cursor position, the cursor on the next prompt is accidentally on
     # the next line.  If this is fixed, uncomment the following line.
-    #if max_offset < offset + expandsize:
-    #    rows, max_offset, orig_posize = _expand_console_buffer(
+    # if max_offset < offset + expandsize:
+    #     rows, max_offset, orig_posize = _expand_console_buffer(
     #                                        cols, max_offset, expandsize,
     #                                        orig_posize, fd)
-    #    winutils.set_console_cursor_position(x, y, fd=fd)
+    #     winutils.set_console_cursor_position(x, y, fd=fd)
     while True:
         posize = winutils.get_position_size(fd)
         offset = (cols*y) + x
@@ -833,7 +838,10 @@ class PopenThread(threading.Thread):
         new[LFLAG] |= termios.ECHO | termios.ICANON
         new[CC][termios.VMIN] = 1
         new[CC][termios.VTIME] = 0
-        termios.tcsetattr(self.stdin_fd, termios.TCSANOW, new)
+        try:
+            termios.tcsetattr(self.stdin_fd, termios.TCSANOW, new)
+        except termios.error:
+            pass
 
     #
     # Dispatch methods
@@ -865,6 +873,11 @@ class PopenThread(threading.Thread):
         """Process return code."""
         return self.proc.returncode
 
+    @returncode.setter
+    def returncode(self, value):
+        """Process return code."""
+        self.proc.returncode = value
+
     @property
     def signal(self):
         """Process signal, or None."""
@@ -875,6 +888,11 @@ class PopenThread(threading.Thread):
                 s = (-1*rtn, rtn < 0 if ON_WINDOWS else os.WCOREDUMP(rtn))
         return s
 
+    @signal.setter
+    def signal(self, value):
+        """Process signal, or None."""
+        self.proc.signal = value
+
     def send_signal(self, signal):
         """Dispatches to Popen.send_signal()."""
         dt = 0.0
@@ -1221,6 +1239,7 @@ class ProcProxyThread(threading.Thread):
         self.stdout = stdout
         self.stderr = stderr
         self.env = env or builtins.__xonsh_env__
+        self._interrupted = False
 
         if ON_WINDOWS:
             if self.p2cwrite != -1:
@@ -1248,9 +1267,19 @@ class ProcProxyThread(threading.Thread):
             if universal_newlines:
                 self.stderr = io.TextIOWrapper(self.stderr)
 
+        # Set some signal handles, if we can. Must come before process
+        # is started to prevent deadlock on windows
+        self.old_int_handler = None
+        if on_main_thread():
+            self.old_int_handler = signal.signal(signal.SIGINT,
+                                                 self._signal_int)
+        # start up the proc
         super().__init__()
         self.start()
 
+    def __del__(self):
+        self._restore_sigint()
+
     def run(self):
         """Set up input/output streams and execute the child function in a new
         thread.  This is part of the `threading.Thread` interface and should
@@ -1354,8 +1383,41 @@ class ProcProxyThread(threading.Thread):
     def wait(self, timeout=None):
         """Waits for the process to finish and returns the return code."""
         self.join()
+        self._restore_sigint()
         return self.returncode
 
+    #
+    # SIGINT handler
+    #
+
+    def _signal_int(self, signum, frame):
+        """Signal handler for SIGINT - Ctrl+C may have been pressed."""
+        # check if we have already be interrupted to prevent infintie recurrsion
+        if self._interrupted:
+            return
+        self._interrupted = True
+        # close file handles here to stop an processes piped to us.
+        handles = (self.p2cread, self.p2cwrite, self.c2pread, self.c2pwrite,
+                   self.errread, self.errwrite)
+        for handle in handles:
+            safe_fdclose(handle)
+        if self.poll() is not None:
+            self._restore_sigint(frame=frame)
+        if on_main_thread():
+            signal.pthread_kill(threading.get_ident(), signal.SIGINT)
+
+    def _restore_sigint(self, frame=None):
+        old = self.old_int_handler
+        if old is not None:
+            if on_main_thread():
+                signal.signal(signal.SIGINT, old)
+            self.old_int_handler = None
+        if frame is not None:
+            if old is not None and old is not self._signal_int:
+                old(signal.SIGINT, frame)
+        if self._interrupted:
+            self.returncode = 1
+
     # The code below (_get_devnull, _get_handles, and _make_inheritable) comes
     # from subprocess.py in the Python 3.4.2 Standard Library
     def _get_devnull(self):
@@ -1539,7 +1601,11 @@ class ProcProxy(object):
         if self.stdin is None:
             stdin = None
         else:
-            stdin = io.TextIOWrapper(self.stdin, encoding=enc, errors=err)
+            if isinstance(self.stdin, int):
+                inbuf = io.open(self.stdin, 'rb', -1)
+            else:
+                inbuf = self.stdin
+            stdin = io.TextIOWrapper(inbuf, encoding=enc, errors=err)
         stdout = self._pick_buf(self.stdout, sys.stdout, enc, err)
         stderr = self._pick_buf(self.stderr, sys.stderr, enc, err)
         # run the actual function
@@ -1578,26 +1644,6 @@ class ProcProxy(object):
         return getattr(self, name)
 
 
-def unthreadable(f):
-    """Decorator that specifies that a callable alias should be run only
-    on the main thread process. This is often needed for debuggers and profilers.
-    """
-    f.__xonsh_threadable__ = False
-    return f
-
-
-foreground = unthreadable
-
-
-def uncapturable(f):
-    """Decorator that specifies that a callable alias should not be run with
-    any capturing. This is often needed if the alias call interactive subprocess,
-    like pagers and text editors.
-    """
-    f.__xonsh_capturable__ = False
-    return f
-
-
 @lazyobject
 def SIGNAL_MESSAGES():
     sm = {
@@ -1634,6 +1680,17 @@ def safe_readable(handle):
     return status
 
 
+def update_fg_process_group(pipeline_group, background):
+    if background:
+        return False
+    if not ON_POSIX:
+        return False
+    env = builtins.__xonsh_env__
+    if not env.get('XONSH_INTERACTIVE'):
+        return False
+    return give_terminal_to(pipeline_group)
+
+
 class CommandPipeline:
     """Represents a subprocess-mode command pipeline."""
 
@@ -1644,18 +1701,12 @@ class CommandPipeline:
 
     nonblocking = (io.BytesIO, NonBlockingFDReader, ConsoleParallelReader)
 
-    def __init__(self, specs, procs, starttime=None, captured=False):
+    def __init__(self, specs):
         """
         Parameters
         ----------
         specs : list of SubprocSpec
             Process sepcifications
-        procs : list of Popen-like
-            Process objects.
-        starttime : floats or None, optional
-            Start timestamp.
-        captured : bool or str, optional
-            Flag for whether or not the command should be captured.
 
         Attributes
         ----------
@@ -1673,18 +1724,38 @@ class CommandPipeline:
             A string of the standard error.
         lines : list of str
             The output lines
+        starttime : floats or None
+            Pipeline start timestamp.
         """
-        self.procs = procs
-        self.proc = procs[-1]
+        self.starttime = None
+        self.ended = False
+        self.procs = []
         self.specs = specs
         self.spec = specs[-1]
-        self.starttime = starttime or time.time()
-        self.captured = captured
-        self.ended = False
+        self.captured = specs[-1].captured
         self.input = self._output = self.errors = self.endtime = None
         self._closed_handle_cache = {}
         self.lines = []
         self._stderr_prefix = self._stderr_postfix = None
+        self.term_pgid = None
+
+        background = self.spec.background
+        pipeline_group = None
+        for spec in specs:
+            if self.starttime is None:
+                self.starttime = time.time()
+            try:
+                proc = spec.run(pipeline_group=pipeline_group)
+            except XonshError:
+                self._return_terminal()
+                raise
+            if proc.pid and pipeline_group is None and not spec.is_proxy and \
+                    self.captured != 'object':
+                pipeline_group = proc.pid
+                if update_fg_process_group(pipeline_group, background):
+                    self.term_pgid = pipeline_group
+            self.procs.append(proc)
+        self.proc = self.procs[-1]
 
     def __repr__(self):
         s = self.__class__.__name__ + '('
@@ -1724,7 +1795,8 @@ class CommandPipeline:
             stdout = stdout.buffer
         if stdout is not None and not isinstance(stdout, self.nonblocking):
             stdout = NonBlockingFDReader(stdout.fileno(), timeout=timeout)
-        if not stdout or not safe_readable(stdout):
+        if not stdout or self.captured == 'stdout' or not safe_readable(stdout) or \
+                not spec.threadable:
             # we get here if the process is not threadable or the
             # class is the real Popen
             PrevProcCloser(pipeline=self)
@@ -1734,6 +1806,15 @@ class CommandPipeline:
                 self._endtime()
                 if self.captured == 'object':
                     self.end(tee_output=False)
+                elif self.captured == 'hiddenobject' and stdout:
+                    b = stdout.read()
+                    lines = b.splitlines(keepends=True)
+                    yield from lines
+                    self.end(tee_output=False)
+                elif self.captured == 'stdout':
+                    b = stdout.read()
+                    s = self._decode_uninew(b, universal_newlines=True)
+                    self.lines = s.splitlines(keepends=True)
             raise StopIteration
         # get the correct stderr
         stderr = proc.stderr
@@ -1817,7 +1898,8 @@ class CommandPipeline:
 
     def tee_stdout(self):
         """Writes the process stdout to the output variable, line-by-line, and
-        yields each line.
+        yields each line. This may optionally accept lines (in bytes) to iterate
+        over, in which case it does not call iterraw().
         """
         env = builtins.__xonsh_env__
         enc = env.get('XONSH_ENCODING')
@@ -1880,7 +1962,7 @@ class CommandPipeline:
         else:
             self.errors += s
 
-    def _decode_uninew(self, b):
+    def _decode_uninew(self, b, universal_newlines=None):
         """Decode bytes into a str and apply universal newlines as needed."""
         if not b:
             return ''
@@ -1890,7 +1972,7 @@ class CommandPipeline:
                          errors=env.get('XONSH_ENCODING_ERRORS'))
         else:
             s = b
-        if self.spec.universal_newlines:
+        if universal_newlines or self.spec.universal_newlines:
             s = s.replace('\r\n', '\n').replace('\r', '\n')
         return s
 
@@ -1899,11 +1981,20 @@ class CommandPipeline:
     #
 
     def end(self, tee_output=True):
-        """Waits for the command to complete and then runs any closing and
-        cleanup procedures that need to be run.
+        """
+        End the pipeline, return the controlling terminal if needed.
+
+        Main things done in self._end().
         """
         if self.ended:
             return
+        self._end(tee_output=tee_output)
+        self._return_terminal()
+
+    def _end(self, tee_output):
+        """Waits for the command to complete and then runs any closing and
+        cleanup procedures that need to be run.
+        """
         if tee_output:
             for _ in self.tee_stdout():
                 pass
@@ -1918,6 +2009,28 @@ class CommandPipeline:
         self.ended = True
         self._raise_subproc_error()
 
+    def _return_terminal(self):
+        if ON_WINDOWS or not ON_POSIX:
+            return
+        pgid = os.getpgid(0)
+        if self.term_pgid is None or pgid == self.term_pgid:
+            return
+        if give_terminal_to(pgid):  # if gave term succeed
+            self.term_pgid = pgid
+            if hasattr(builtins, '__xonsh_shell__'):
+                # restoring sanity could probably be called whenever we return
+                # control to the shell. But it only seems to matter after a
+                # ^Z event. This *has* to be called after we give the terminal
+                # back to the shell.
+                builtins.__xonsh_shell__.shell.restore_tty_sanity()
+
+    def resume(self, job, tee_output=True):
+        self.ended = False
+        if give_terminal_to(job['pgrp']):
+            self.term_pgid = job['pgrp']
+        _continue(job)
+        self.end(tee_output=tee_output)
+
     def _endtime(self):
         """Sets the closing timestamp if it hasn't been already."""
         if self.endtime is None:

xonsh/ptk/completer.py

@@ -5,6 +5,7 @@ import builtins
 
 from prompt_toolkit.layout.dimension import LayoutDimension
 from prompt_toolkit.completion import Completer, Completion
+from prompt_toolkit.auto_suggest import AutoSuggestFromHistory
 
 
 class PromptToolkitCompleter(Completer):
@@ -13,51 +14,75 @@ class PromptToolkitCompleter(Completer):
     It just redirects requests to normal Xonsh completer.
     """
 
-    def __init__(self, completer, ctx):
-        """Takes instance of xonsh.completer.Completer and dict with context."""
+    def __init__(self, completer, ctx, shell):
+        """Takes instance of xonsh.completer.Completer, the xonsh execution
+        context, and the shell instance itself.
+        """
         self.completer = completer
         self.ctx = ctx
+        self.shell = shell
+        self.hist_suggester = AutoSuggestFromHistory()
 
     def get_completions(self, document, complete_event):
         """Returns a generator for list of completions."""
-
+        env = builtins.__xonsh_env__
+        should_complete = (
+            complete_event.completion_requested or
+            env.get('UPDATE_COMPLETIONS_ON_KEYPRESS')
+            )
         #  Only generate completions when the user hits tab.
-        if complete_event.completion_requested:
-            if self.completer is None:
-                yield from []
+        if not should_complete or self.completer is None:
+            return
+        # generate actual completions
+        line = document.current_line.lstrip()
+        line_ex = builtins.aliases.expand_alias(line)
+
+        endidx = document.cursor_position_col
+        begidx = (line[:endidx].rfind(' ') + 1
+                  if line[:endidx].rfind(' ') >= 0 else 0)
+        prefix = line[begidx:endidx]
+        expand_offset = len(line_ex) - len(line)
+        # get normal completions
+        completions, l = self.completer.complete(prefix, line_ex,
+                                                 begidx + expand_offset,
+                                                 endidx + expand_offset,
+                                                 self.ctx)
+        # completions from auto suggest
+        if env.get('AUTO_SUGGEST'):
+            sug_comp = self.suggestion_completion(document, line)
+            if sug_comp is None:
+                pass
+            elif len(completions) == 0:
+                completions = (sug_comp,)
             else:
-                line = document.current_line.lstrip()
-                line_ex = builtins.aliases.expand_alias(line)
+                completions = (sug_comp,) + completions
+        # reserve space, if needed.
+        if len(completions) <= 1:
+            pass
+        elif len(os.path.commonprefix(completions)) <= len(prefix):
+            self.reserve_space()
+        # Find common prefix (strip quoting)
+        c_prefix = os.path.commonprefix([a.strip('\'"') for a in completions])
+        # Find last split symbol, do not trim the last part
+        while c_prefix:
+            if c_prefix[-1] in r'/\.:@,':
+                break
+            c_prefix = c_prefix[:-1]
+        # yield completions
+        for comp in completions:
+            # do not display quote
+            disp = comp.strip('\'"')[len(c_prefix):]
+            yield Completion(comp, -l, display=disp)
 
-                endidx = document.cursor_position_col
-                begidx = (line[:endidx].rfind(' ') + 1
-                          if line[:endidx].rfind(' ') >= 0 else 0)
-                prefix = line[begidx:endidx]
-                expand_offset = len(line_ex) - len(line)
-
-                completions, l = self.completer.complete(prefix,
-                                                         line_ex,
-                                                         begidx + expand_offset,
-                                                         endidx + expand_offset,
-                                                         self.ctx)
-                if len(completions) <= 1:
-                    pass
-                elif len(os.path.commonprefix(completions)) <= len(prefix):
-                    self.reserve_space()
-
-                # Find common prefix (strip quoting)
-                c_prefix = os.path.commonprefix([a.strip('\'"')
-                                                for a in completions])
-                # Find last split symbol, do not trim the last part
-                while c_prefix:
-                    if c_prefix[-1] in r'/\.:@,':
-                        break
-                    c_prefix = c_prefix[:-1]
-
-                for comp in completions:
-                    # do not display quote
-                    disp = comp.strip('\'"')[len(c_prefix):]
-                    yield Completion(comp, -l, display=disp)
+    def suggestion_completion(self, document, line):
+        """Provides a completion based on the current auto-suggestion."""
+        cli = self.shell.prompter.cli
+        sug = self.hist_suggester.get_suggestion(cli, cli.current_buffer, document)
+        if sug is None:
+            return None
+        comp, _, _ = sug.text.partition(' ')
+        _, _, prev = line.rpartition(' ')
+        return prev + comp
 
     def reserve_space(self):
         cli = builtins.__xonsh_shell__.shell.prompter.cli

xonsh/ptk/shell.py

@@ -10,7 +10,7 @@ from prompt_toolkit.shortcuts import print_tokens
 from prompt_toolkit.styles import PygmentsStyle, style_from_dict
 
 from xonsh.base_shell import BaseShell
-from xonsh.tools import print_exception
+from xonsh.tools import print_exception, carriage_return
 from xonsh.ptk.completer import PromptToolkitCompleter
 from xonsh.ptk.history import PromptToolkitHistory
 from xonsh.ptk.key_bindings import load_xonsh_bindings
@@ -36,9 +36,10 @@ class PromptToolkitShell(BaseShell):
 
     def __init__(self, **kwargs):
         super().__init__(**kwargs)
+        self._first_prompt = True
         self.prompter = Prompter()
         self.history = PromptToolkitHistory()
-        self.pt_completer = PromptToolkitCompleter(self.completer, self.ctx)
+        self.pt_completer = PromptToolkitCompleter(self.completer, self.ctx, self)
         key_bindings_manager_args = {
             'enable_auto_suggest_bindings': True,
             'enable_search': True,
@@ -70,6 +71,10 @@ class PromptToolkitShell(BaseShell):
         auto_suggest = auto_suggest if env.get('AUTO_SUGGEST') else None
         completions_display = env.get('COMPLETIONS_DISPLAY')
         multicolumn = (completions_display == 'multi')
+        complete_while_typing = env.get('UPDATE_COMPLETIONS_ON_KEYPRESS')
+        if complete_while_typing:
+            # PTK requires history search to be none when completing while typing
+            enable_history_search = False
         if HAS_PYGMENTS:
             self.styler.style_name = env.get('XONSH_COLOR_STYLE')
         completer = None if completions_display == 'none' else self.pt_completer
@@ -100,6 +105,7 @@ class PromptToolkitShell(BaseShell):
                     'reserve_space_for_menu': 0,
                     'key_bindings_registry': self.key_bindings_manager.registry,
                     'display_completions_in_columns': multicolumn,
+                    'complete_while_typing': complete_while_typing,
                     }
             if builtins.__xonsh_env__.get('COLOR_INPUT'):
                 if HAS_PYGMENTS:
@@ -107,8 +113,9 @@ class PromptToolkitShell(BaseShell):
                     prompt_args['style'] = PygmentsStyle(pyghooks.xonsh_style_proxy(self.styler))
                 else:
                     prompt_args['style'] = style_from_dict(DEFAULT_STYLE_DICT)
-
+            events.on_pre_prompt.fire()
             line = self.prompter.prompt(**prompt_args)
+            events.on_post_prompt.fire()
         return line
 
     def _push(self, line):
@@ -163,6 +170,9 @@ class PromptToolkitShell(BaseShell):
         except Exception:  # pylint: disable=broad-except
             print_exception()
         toks = partial_color_tokenize(p)
+        if self._first_prompt:
+            carriage_return()
+            self._first_prompt = False
         self.settitle()
         return toks
 

xonsh/pytest_plugin.py

@@ -5,11 +5,12 @@ import importlib
 from traceback import format_list, extract_tb
 
 import pytest
-from xonsh.imphooks import install_hook
+
+from xonsh.imphooks import install_import_hooks
 
 
 def pytest_configure(config):
-    install_hook()
+    install_import_hooks()
 
 
 def pytest_collection_modifyitems(items):

xonsh/readline_shell.py

@@ -27,9 +27,10 @@ from xonsh.ansi_colors import (ansi_partial_color_format, ansi_color_style_names
                                ansi_color_style)
 from xonsh.prompt.base import multiline_prompt
 from xonsh.tools import (print_exception, check_for_partial_string, to_bool,
-                         columnize)
+                         columnize, carriage_return)
 from xonsh.platform import ON_WINDOWS, ON_CYGWIN, ON_DARWIN, ON_POSIX
-from xonsh.lazyimps import pygments, pyghooks
+from xonsh.lazyimps import pygments, pyghooks, winutils
+from xonsh.events import events
 
 readline = None
 RL_COMPLETION_SUPPRESS_APPEND = RL_LIB = RL_STATE = None
@@ -125,12 +126,16 @@ def setup_readline():
         inputrc_name = os.path.join(os.path.expanduser('~'), inputrc_name)
     if (not ON_WINDOWS) and (not os.path.isfile(inputrc_name)):
         inputrc_name = '/etc/inputrc'
+    if ON_WINDOWS:
+        winutils.enable_virtual_terminal_processing()
     if os.path.isfile(inputrc_name):
         try:
             readline.read_init_file(inputrc_name)
         except Exception:
             # this seems to fail with libedit
             print_exception('xonsh: could not load readline default init file.')
+    # properly reset intput typed before the first prompt
+    readline.set_startup_hook(carriage_return)
 
 
 def teardown_readline():
@@ -229,6 +234,10 @@ def rl_on_new_line():
         func = getattr(RL_LIB, name, None)
         if func is not None:
             break
+    else:
+        def print_for_newline():
+            print()
+        func = print_for_newline
     return func
 
 
@@ -275,7 +284,9 @@ class ReadlineShell(BaseShell, cmd.Cmd):
             except ImportError:
                 store_in_history = True
             pos = readline.get_current_history_length() - 1
+        events.on_pre_prompt.fire()
         rtn = input(self.prompt)
+        events.on_post_prompt.fire()
         if not store_in_history and pos >= 0:
             readline.remove_history_item(pos)
         return rtn

xonsh/shell.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 """The xonsh shell"""
-import os
 import sys
 import random
 import time
@@ -8,12 +7,9 @@ import difflib
 import builtins
 import warnings
 
-from xonsh.xontribs import update_context, prompt_xontrib_install
-from xonsh.environ import xonshrc_context
-from xonsh.execer import Execer
 from xonsh.platform import (best_shell_type, has_prompt_toolkit,
                             ptk_version_is_supported)
-from xonsh.tools import XonshError, to_bool_or_int, print_exception
+from xonsh.tools import XonshError, print_exception
 from xonsh.events import events
 import xonsh.history.main as xhm
 
@@ -39,6 +35,25 @@ events.doc('on_postcommand', """
 on_postcommand(cmd: str, rtn: int, out: str or None, ts: list) -> None
 
 Fires just after a command is executed. The arguments are the same as history.
+
+Parameters:
+
+* ``cmd``: The command that was executed (after transformation)
+* ``rtn``: The result of the command executed (``0`` for success)
+* ``out``: If xonsh stores command output, this is the output
+* ``ts``: Timestamps, in the order of ``[starting, ending]``
+""")
+
+events.doc('on_pre_prompt', """
+on_first_prompt() -> None
+
+Fires just before the prompt is shown
+""")
+
+events.doc('on_post_prompt', """
+on_first_prompt() -> None
+
+Fires just after the prompt returns
 """)
 
 
@@ -78,11 +93,12 @@ class Shell(object):
     readline version of shell should be used.
     """
 
-    def __init__(self, ctx=None, shell_type=None, config=None, rc=None,
-                 **kwargs):
+    def __init__(self, execer, ctx=None, shell_type=None, **kwargs):
         """
         Parameters
         ----------
+        execer : Execer
+            An execer instance capable of running xonsh code.
         ctx : Mapping, optional
             The execution context for the shell (e.g. the globals namespace).
             If none, this is computed by loading the rc files. If not None,
@@ -91,17 +107,10 @@ class Shell(object):
         shell_type : str, optional
             The shell type to start, such as 'readline', 'prompt_toolkit',
             or 'random'.
-        config : str, optional
-            Path to configuration file.
-        rc : list of str, optional
-            Sequence of paths to run control files.
         """
-        self.login = kwargs.get('login', True)
+        self.execer = execer
+        self.ctx = {} if ctx is None else ctx
         self.stype = shell_type
-        self._init_environ(ctx, config, rc,
-                           kwargs.get('scriptcache', True),
-                           kwargs.get('cacheall', False))
-
         env = builtins.__xonsh_env__
         # build history backend before creating shell
         builtins.__xonsh_history__ = hist = xhm.construct_history(
@@ -149,26 +158,3 @@ class Shell(object):
     def __getattr__(self, attr):
         """Delegates calls to appropriate shell instance."""
         return getattr(self.shell, attr)
-
-    def _init_environ(self, ctx, config, rc, scriptcache, cacheall):
-        self.ctx = {} if ctx is None else ctx
-        debug = to_bool_or_int(os.getenv('XONSH_DEBUG', '0'))
-        self.execer = Execer(config=config, login=self.login, xonsh_ctx=self.ctx,
-                             debug_level=debug)
-        self.execer.scriptcache = scriptcache
-        self.execer.cacheall = cacheall
-        if self.stype != 'none' or self.login:
-            # load xontribs from config file
-            names = builtins.__xonsh_config__.get('xontribs', ())
-            for name in names:
-                update_context(name, ctx=self.ctx)
-            if getattr(update_context, 'bad_imports', None):
-                prompt_xontrib_install(update_context.bad_imports)
-                del update_context.bad_imports
-            # load run control files
-            env = builtins.__xonsh_env__
-            rc = env.get('XONSHRC') if rc is None else rc
-            events.on_pre_rc.fire()
-            self.ctx.update(xonshrc_context(rcfiles=rc, execer=self.execer, initial=self.ctx))
-            events.on_post_rc.fire()
-        self.ctx['__name__'] = '__main__'

xonsh/style_tools.py

@@ -8,20 +8,52 @@ from xonsh.lazyasd import LazyObject
 
 class _TokenType(tuple):
     """
-    This class was forked from the mainline prompt-toolkit repo.
-    Copyright (c) 2014, Jonathan Slenders, All rights reserved.
+    Forked from the pygments project
+    https://bitbucket.org/birkenfeld/pygments-main
+    Copyright (c) 2006-2017 by the respective authors, All rights reserved.
+    See https://bitbucket.org/birkenfeld/pygments-main/raw/05818a4ef9891d9ac22c851f7b3ea4b4fce460ab/AUTHORS
     """
+    parent = None
+
+    def split(self):
+        buf = []
+        node = self
+        while node is not None:
+            buf.append(node)
+            node = node.parent
+        buf.reverse()
+        return buf
+
+    def __init__(self, *args):
+        # no need to call super.__init__
+        self.subtypes = set()
+
+    def __contains__(self, val):
+        return self is val or (
+            type(val) is self.__class__ and
+            val[:len(self)] == self
+        )
+
     def __getattr__(self, val):
         if not val or not val[0].isupper():
             return tuple.__getattribute__(self, val)
-
         new = _TokenType(self + (val,))
         setattr(self, val, new)
+        self.subtypes.add(new)
+        new.parent = self
         return new
 
     def __repr__(self):
         return 'Token' + (self and '.' or '') + '.'.join(self)
 
+    def __copy__(self):
+        # These instances are supposed to be singletons
+        return self
+
+    def __deepcopy__(self, memo):
+        # These instances are supposed to be singletons
+        return self
+
 RE_BACKGROUND = LazyObject(lambda: re.compile('(BG#|BGHEX|BACKGROUND)'),
                            globals(), 'RE_BACKGROUND')
 

xonsh/timings.py

@@ -7,6 +7,7 @@ The following time_it alias and Timer was forked from the IPython project:
 * Copyright (c) 2001, Janko Hauser <jhauser@zscout.de>
 * Copyright (c) 2001, Nathaniel Gray <n8gray@caltech.edu>
 """
+import os
 import gc
 import sys
 import math
@@ -16,6 +17,8 @@ import builtins
 import itertools
 
 from xonsh.lazyasd import lazyobject, lazybool
+from xonsh.events import events
+from xonsh.platform import ON_WINDOWS
 
 
 @lazybool
@@ -223,3 +226,92 @@ def timeit_alias(args, stdin=None):
         if tc > tc_min:
             print("Compiler time: {0:.2f} s".format(tc))
     return
+
+
+_timings = {'start': clock() if ON_WINDOWS else 0.0}
+
+
+def setup_timings():
+    global _timings
+    if '--timings' in sys.argv:
+        events.doc('on_timingprobe', """
+        on_timingprobe(name: str) -> None
+
+        Fired to insert some timings into the startuptime list
+        """)
+
+        @events.on_timingprobe
+        def timing_on_timingprobe(name, **kw):
+            global _timings
+            _timings[name] = clock()
+
+        @events.on_post_cmdloop
+        def timing_on_post_cmdloop(**kw):
+            global _timings
+            _timings['on_post_cmdloop'] = clock()
+
+        @events.on_post_init
+        def timing_on_post_init(**kw):
+            global _timings
+            _timings['on_post_init'] = clock()
+
+        @events.on_post_rc
+        def timing_on_post_rc(**kw):
+            global _timings
+            _timings['on_post_rc'] = clock()
+
+        @events.on_postcommand
+        def timing_on_postcommand(**kw):
+            global _timings
+            _timings['on_postcommand'] = clock()
+
+        @events.on_pre_cmdloop
+        def timing_on_pre_cmdloop(**kw):
+            global _timings
+            _timings['on_pre_cmdloop'] = clock()
+
+        @events.on_pre_rc
+        def timing_on_pre_rc(**kw):
+            global _timings
+            _timings['on_pre_rc'] = clock()
+
+        @events.on_precommand
+        def timing_on_precommand(**kw):
+            global _timings
+            _timings['on_precommand'] = clock()
+
+        @events.on_ptk_create
+        def timing_on_ptk_create(**kw):
+            global _timings
+            _timings['on_ptk_create'] = clock()
+
+        @events.on_chdir
+        def timing_on_chdir(**kw):
+            global _timings
+            _timings['on_chdir'] = clock()
+
+        @events.on_post_prompt
+        def timing_on_post_prompt(**kw):
+            global _timings
+            _timings = {'on_post_prompt': clock()}
+
+        @events.on_pre_prompt
+        def timing_on_pre_prompt(**kw):
+            global _timings
+            _timings['on_pre_prompt'] = clock()
+            times = list(_timings.items())
+            times = sorted(times, key=lambda x: x[1])
+            width = max(len(s) for s, _ in times) + 2
+            header_format = '|{{:<{}}}|{{:^11}}|{{:^11}}|'.format(width)
+            entry_format = '|{{:<{}}}|{{:^11.3f}}|{{:^11.3f}}|'.format(width)
+            sepline = '|{}|{}|{}|'.format('-'*width, '-'*11, '-'*11)
+            # Print result table
+            print(' Debug level: {}'.format(os.getenv('XONSH_DEBUG', 'Off')))
+            print(sepline)
+            print(header_format.format('Event name', 'Time (s)', 'Delta (s)'))
+            print(sepline)
+            prevtime = tstart = times[0][1]
+            for name, ts in times:
+                print(entry_format.format(name, ts - tstart, ts - prevtime))
+                prevtime = ts
+            print(sepline)

xonsh/tokenize.py

@@ -249,7 +249,6 @@ Special = group(r'\r?\n', r'\.\.\.', r'[:;.,@]')
 Funny = group(Operator, Bracket, Special)
 
 PlainToken = group(IORedirect, Number, Funny, String, Name_RE, SearchPath)
-Token = Ignore + PlainToken
 
 # First (or only) line of ' or " string.
 ContStr = group(StringPrefix + r"'[^\n'\\]*(?:\\.[^\n'\\]*)*" +

xonsh/tools.py

@@ -346,6 +346,44 @@ def subproc_toks(line, mincol=-1, maxcol=None, lexer=None, returnline=False):
     return rtn
 
 
+def get_logical_line(lines, idx):
+    """Returns a single logical line (i.e. one without line continuations)
+    from a list of lines.  This line should begin at index idx. This also
+    returns the number of physical lines the logical line spans. The lines
+    should not contain newlines
+    """
+    n = 1
+    nlines = len(lines)
+    line = lines[idx]
+    while line.endswith('\\') and idx < nlines:
+        n += 1
+        idx += 1
+        line = line[:-1] + lines[idx]
+    return line, n
+
+
+def replace_logical_line(lines, logical, idx, n):
+    """Replaces lines at idx that may end in line continuation with a logical
+    line that spans n lines.
+    """
+    if n == 1:
+        lines[idx] = logical
+        return
+    space = ' '
+    for i in range(idx, idx+n-1):
+        a = len(lines[i])
+        b = logical.find(space, a-1)
+        if b < 0:
+            # no space found
+            lines[i] = logical
+            logical = ''
+        else:
+            # found space to split on
+            lines[i] = logical[:b] + '\\'
+            logical = logical[b:]
+    lines[idx+n-1] = logical
+
+
 def is_balanced(expr, ltok, rtok):
     """Determines whether an expression has unbalanced opening and closing tokens."""
     lcnt = expr.count(ltok)
@@ -792,8 +830,11 @@ def on_main_thread():
     return threading.current_thread() is threading.main_thread()
 
 
+_DEFAULT_SENTINEL = object()
+
+
 @contextlib.contextmanager
-def swap(namespace, name, value, default=NotImplemented):
+def swap(namespace, name, value, default=_DEFAULT_SENTINEL):
     """Swaps a current variable name in a namespace for another value, and then
     replaces it when the context is exited.
     """
@@ -805,6 +846,21 @@ def swap(namespace, name, value, default=NotImplemented):
     else:
         setattr(namespace, name, old)
 
+
+@contextlib.contextmanager
+def swap_values(d, updates, default=_DEFAULT_SENTINEL):
+    """Updates a dictionary (or other mapping) with values from another mapping,
+    and then restores the original mapping when the context is exited.
+    """
+    old = {k: d.get(k, default) for k in updates}
+    d.update(updates)
+    yield
+    for k, v in old.items():
+        if v is default and k in d:
+            del d[k]
+        else:
+            d[k] = v
+
 #
 # Validators and converters
 #
@@ -1773,3 +1829,26 @@ def columnize(elems, width=80, newline='\n'):
     lines = [row_t.format(row=row, w=colwidths) for row in
              itertools.zip_longest(*data, fillvalue='')]
     return lines
+
+
+def unthreadable(f):
+    """Decorator that specifies that a callable alias should be run only
+    on the main thread process. This is often needed for debuggers and
+    profilers.
+    """
+    f.__xonsh_threadable__ = False
+    return f
+
+
+def uncapturable(f):
+    """Decorator that specifies that a callable alias should not be run with
+    any capturing. This is often needed if the alias call interactive
+    subprocess, like pagers and text editors.
+    """
+    f.__xonsh_capturable__ = False
+    return f
+
+
+def carriage_return():
+    """Writes a carriage return to stdout, and nothing else."""
+    print('\r', flush=True, end='')

xonsh/tracer.py

@@ -13,9 +13,9 @@ from xonsh.platform import HAS_PYGMENTS
 from xonsh.tools import DefaultNotGiven, print_color, normabspath, to_bool
 from xonsh.inspectors import find_file, getouterframes
 from xonsh.lazyimps import pygments, pyghooks
+from xonsh.proc import STDOUT_CAPTURE_KINDS
 import xonsh.prompt.cwd as prompt
 
-
 terminal = LazyObject(lambda: importlib.import_module(
                                 'pygments.formatters.terminal'),
                       globals(), 'terminal')
@@ -45,6 +45,14 @@ class TracerType(object):
         for f in set(self.files):
             self.stop(f)
 
+    def color_output(self, usecolor):
+        """Specify whether or not the tracer output should be colored."""
+        # we have to use a function to set usecolor because of the way that
+        # lazyasd works. Namely, it cannot dispatch setattr to the target
+        # object without being unable to access its own __dict__. This makes
+        # setting an atter look like getting a function.
+        self.usecolor = usecolor
+
     def start(self, filename):
         """Starts tracing a file."""
         files = self.files
@@ -109,6 +117,10 @@ def tracer_format_line(fname, lineno, line, color=True, lexer=None, formatter=No
     tokens = pyghooks.partial_color_tokenize(cline)
     lexer = lexer or pyghooks.XonshLexer()
     tokens += pygments.lex(line, lexer=lexer)
+    if tokens[-1][1] == '\n':
+        del tokens[-1]
+    elif tokens[-1][1].endswith('\n'):
+        tokens[-1] = (tokens[-1][0], tokens[-1][1].rstrip())
     return tokens
 
 
@@ -157,7 +169,7 @@ def _off(ns, args):
 
 def _color(ns, args):
     """Manages color action for tracer CLI."""
-    tracer.usecolor = ns.toggle
+    tracer.color_output(ns.toggle)
 
 
 @functools.lru_cache(1)
@@ -194,8 +206,11 @@ _TRACER_MAIN_ACTIONS = {
     }
 
 
-def tracermain(args=None):
+def tracermain(args=None, stdin=None, stdout=None, stderr=None, spec=None):
     """Main function for tracer command-line interface."""
     parser = _tracer_create_parser()
     ns = parser.parse_args(args)
+    usecolor = ((spec.captured not in STDOUT_CAPTURE_KINDS) and
+                sys.stdout.isatty())
+    tracer.color_output(usecolor)
     return _TRACER_MAIN_ACTIONS[ns.action](ns, args)

xonsh/winutils.py

@@ -249,6 +249,14 @@ def set_console_mode(mode, fd=1):
     SetConsoleMode(hcon, mode)
 
 
+def enable_virtual_terminal_processing():
+    """Enables virtual terminal processing on Windows.
+    This inlcudes ANSI escape sequence interpretation.
+    See http://stackoverflow.com/a/36760881/2312428
+    """
+    SetConsoleMode(GetStdHandle(-11), 7)
+
+
 @lazyobject
 def COORD():
     if platform.has_prompt_toolkit():

xonsh/xontribs.json

@@ -23,6 +23,25 @@
     "its behavior. To see the modifications as they are applied (in unified diff",
     "format), please set ``$XONSH_DEBUG`` to ``2`` or higher."]
  },
+ {"name": "coreutils",
+  "package": "xonsh",
+  "url": "http://xon.sh",
+  "description": [
+    "Additional core utilites that are implemened in xonsh. The current list ",
+    "includes:\n",
+    "\n",
+    "* cat\n",
+    "* echo\n",
+    "* pwd\n",
+    "* tee\n",
+    "* tty",
+    "* yes\n",
+    "\n",
+    "In many cases, these may have a lower performance overhead than the ",
+    "posix command line utility with the same name. This is because these ",
+    "tools avoid the need for a full subprocess call. Additionally, these ",
+    "tools are cross-platform."]
+ },
  {"name": "distributed",
   "package": "xonsh",
   "url": "http://xon.sh",

xonsh/xoreutils/cat.py

@@ -0,0 +1,106 @@
+"""Implements a cat command for xonsh."""
+import os
+
+from xonsh.xoreutils.util import arg_handler
+
+
+def _cat_single_file(opts, fname, stdin, out, err, line_count=1):
+    if fname == '-':
+        f = stdin
+    elif os.path.isdir(fname):
+        print("cat: {}: Is a directory.".format(fname), file=err)
+        return True, line_count
+    elif not os.path.exists(fname):
+        print("cat: No such file or directory: {}".format(fname), file=err)
+        return True, line_count
+    else:
+        f = open(fname, 'rb')
+    sep = os.linesep.encode()
+    last_was_blank = False
+    while True:
+        _r = r = f.readline()
+        if isinstance(_r, str):
+            _r = r = _r.encode()
+        if r == b'':
+            break
+        if r.endswith(sep):
+            _r = _r[:-len(sep)]
+        this_one_blank = _r == b''
+        if last_was_blank and this_one_blank and opts['squeeze_blank']:
+            continue
+        last_was_blank = this_one_blank
+        if (opts['number_all'] or
+                (opts['number_nonblank'] and not this_one_blank)):
+            start = ("%6d " % line_count).encode()
+            _r = start + _r
+            line_count += 1
+        if opts['show_ends']:
+            _r = _r + b'$'
+        try:
+            print(_r.decode('unicode_escape'), flush=True, file=out)
+        except:
+            pass
+    return False, line_count
+
+
+def cat(args, stdin, stdout, stderr):
+    """A cat command for xonsh."""
+    opts = _cat_parse_args(args)
+    if opts is None:
+        print(CAT_HELP_STR, file=stdout)
+        return 0
+
+    line_count = 1
+    errors = False
+    if len(args) == 0:
+        args = ['-']
+    for i in args:
+        o = _cat_single_file(opts, i, stdin, stdout, stderr, line_count)
+        if o is None:
+            return -1
+        _e, line_count = o
+        errors = _e or errors
+
+    return int(errors)
+
+
+def _cat_parse_args(args):
+    out = {'number_nonblank': False, 'number_all': False, 'squeeze_blank': False, 'show_ends': False}
+    if '--help' in args:
+        return
+
+    arg_handler(args, out, '-b', 'number_nonblank', True, '--number-nonblank')
+    arg_handler(args, out, '-n', 'number_all', True, '--number')
+    arg_handler(args, out, '-E', 'show_ends', True, '--show-ends')
+    arg_handler(args, out, '-s', 'squeeze_blank', True, '--squeeze-blank')
+    arg_handler(args, out, '-T', 'show_tabs', True, '--show-tabs')
+
+    return out
+
+
+CAT_HELP_STR = """This version of cat was written in Python for the xonsh project: http://xon.sh
+Based on cat from GNU coreutils: http://www.gnu.org/software/coreutils/
+
+Usage: cat [OPTION]... [FILE]...
+Concatenate FILE(s), or standard input, to standard output.
+
+  -b, --number-nonblank    number nonempty output lines, overrides -n
+  -E, --show-ends          display $ at end of each line
+  -n, --number             number all output lines
+  -s, --squeeze-blank      suppress repeated empty output lines
+  -T, --show-tabs          display TAB characters as ^I
+  -u                       (ignored)
+      --help     display this help and exit
+
+With no FILE, or when FILE is -, read standard input.
+
+Examples:
+  cat f - g  Output f's contents, then standard input, then g's contents.
+  cat        Copy standard input to standard output."""
+
+# NOT IMPLEMENTED:
+#  -A, --show-all           equivalent to -vET
+#  -e                       equivalent to -vE
+#  -t                       equivalent to -vT
+#  -v, --show-nonprinting   use ^ and M- notation, except for LFD and TAB
+#      --version  output version information and exit"""

xonsh/xoreutils/echo.py

@@ -0,0 +1,44 @@
+"""Implements a simple echo command for xonsh."""
+
+
+def echo(args, stdin, stdout, stderr):
+    """A simple echo command."""
+    opts = _echo_parse_args(args)
+    if opts is None:
+        return
+    if opts['help']:
+        print(ECHO_HELP, file=stdout)
+        return 0
+    ender = opts['end']
+    args = map(str, args)
+    if opts['escapes']:
+        args = map(lambda x: x.encode().decode('unicode_escape'), args)
+    print(*args, end=ender, file=stdout)
+
+
+def _echo_parse_args(args):
+    out = {'escapes': False, 'end': '\n'}
+    if '-e' in args:
+        args.remove('-e')
+        out['escapes'] = True
+    if '-E' in args:
+        args.remove('-E')
+        out['escapes'] = False
+    if '-n' in args:
+        args.remove('-n')
+        out['end'] = ''
+    if '-h' in args or '--help' in args:
+        out['help'] = True
+    return out
+
+
+ECHO_HELP = """Usage: echo [OPTIONS]... [STRING]...
+Echo the STRING(s) to standard output.
+
+  -n             do not include the trailing newline
+  -e             enable interpretation of backslash escapes
+  -E             disable interpretation of backslash escapes (default)
+  -h  --help     display this message and exit
+
+This version of echo was written in Python for the xonsh project: http://xon.sh
+Based on echo from GNU coreutils: http://www.gnu.org/software/coreutils/"""

xonsh/xoreutils/pwd.py

@@ -0,0 +1,28 @@
+"""A pwd implementation for xonsh."""
+import os
+
+
+def pwd(args, stdin, stdout, stderr):
+    """A pwd implementation"""
+    e = __xonsh_env__['PWD']
+    if '-h' in args or '--help' in args:
+        print(PWD_HELP, file=stdout)
+        return 0
+    if '-P' in args:
+        e = os.path.realpath(e)
+    print(e, file=stdout)
+    return 0
+
+
+PWD_HELP = """Usage: pwd [OPTION]...
+Print the full filename of the current working directory.
+
+  -P, --physical   avoid all symlinks
+      --help       display this help and exit
+
+This version of pwd was written in Python for the xonsh project: http://xon.sh
+Based on pwd from GNU coreutils: http://www.gnu.org/software/coreutils/"""
+
+
+# Not Implemented
+#   -L, --logical    use PWD from environment, even if it contains symlinks

xonsh/xoreutils/tee.py

@@ -0,0 +1,59 @@
+"""A tee implementation for xonsh."""
+
+
+def tee(args, stdin, stdout, stderr):
+    """A tee command for xonsh."""
+    mode = 'w'
+    if '-a' in args:
+        args.remove('-a')
+        mode = 'a'
+    if '--append' in args:
+        args.remove('--append')
+        mode = 'a'
+    if '--help' in args:
+        print(TEE_HELP, file=stdout)
+        return 0
+    if stdin is None:
+        msg = "tee was not piped stdin, must have input stream to read from."
+        print(msg, file=stderr)
+        return 1
+
+    errors = False
+    files = []
+    for i in args:
+        if i == '-':
+            files.append(stdout)
+        else:
+            try:
+                files.append(open(i, mode))
+            except:
+                print('tee: failed to open {}'.format(i), file=stderr)
+                errors = True
+    files.append(stdout)
+
+    while True:
+        r = stdin.read(1024)
+        if r == '':
+            break
+        for i in files:
+            i.write(r)
+    for i in files:
+        if i != stdout:
+            i.close()
+
+    return int(errors)
+
+
+TEE_HELP = """This version of tee was written in Python for the xonsh project: http://xon.sh
+Based on tee from GNU coreutils: http://www.gnu.org/software/coreutils/
+
+Usage: tee [OPTION]... [FILE]...
+Copy standard input to each FILE, and also to standard output.
+
+  -a, --append              append to the given FILEs, do not overwrite
+      --help     display this help and exit
+
+If a FILE is -, copy again to standard output."""
+
+# NOT IMPLEMENTED:
+#  -i, --ignore-interrupts   ignore interrupt signals

xonsh/xoreutils/tty.py

@@ -0,0 +1,44 @@
+"""A tty implementation for xonsh"""
+import os
+import sys
+
+
+def tty(args, stdin, stdout, stderr):
+    """A tty command for xonsh."""
+    if '--help' in args:
+        print(TTY_HELP, file=stdout)
+        return 0
+    silent = False
+    for i in ('-s', '--silent', '--quiet'):
+        if i in args:
+            silent = True
+            args.remove(i)
+    if len(args) > 0:
+        if not silent:
+            for i in args:
+                print('tty: Invalid option: {}'.format(i), file=stderr)
+            print("Try 'tty --help' for more information", file=stderr)
+        return 2
+    try:
+        fd = stdin.fileno()
+    except:
+        fd = sys.stdin.fileno()
+    if not os.isatty(fd):
+        if not silent:
+            print('not a tty', file=stdout)
+        return 1
+    if not silent:
+        try:
+            print(os.ttyname(fd), file=stdout)
+        except:
+            return 3
+    return 0
+
+TTY_HELP = """Usage: tty [OPTION]...
+Print the file name of the terminal connected to standard input.
+
+  -s, --silent, --quiet   print nothing, only return an exit status
+      --help     display this help and exit
+
+This version of tty was written in Python for the xonsh project: http://xon.sh
+Based on tty from GNU coreutils: http://www.gnu.org/software/coreutils/"""

xonsh/xoreutils/util.py

@@ -0,0 +1,19 @@
+"""Assorted utilities for xonsh core utils."""
+
+
+def arg_handler(args, out, short, key, val, long=None):
+    """A simple argument handler for xoreutils."""
+    if short in args:
+        args.remove(short)
+        if isinstance(key, (list, tuple)):
+            for k in key:
+                out[k] = val
+        else:
+            out[key] = val
+    if long is not None and long in args:
+        args.remove(long)
+        if isinstance(key, (list, tuple)):
+            for k in key:
+                out[k] = val
+        else:
+            out[key] = val

xonsh/xoreutils/yes.py

@@ -0,0 +1,25 @@
+"""An implementation of yes for xonsh."""
+
+
+def yes(args, stdin, stdout, stderr):
+    """A yes command."""
+    if '--help' in args:
+        print(YES_HELP, file=stdout)
+        return 0
+
+    to_print = ["y"] if len(args) == 0 else [str(i) for i in args]
+
+    while True:
+        print(*to_print, file=stdout)
+
+    return 0
+
+
+YES_HELP = """Usage: yes [STRING]...
+  or:  yes OPTION
+Repeatedly output a line with all specified STRING(s), or 'y'.
+
+      --help     display this help and exit
+
+This version of yes was written in Python for the xonsh project: http://xon.sh
+Based on yes from GNU coreutils: http://www.gnu.org/software/coreutils/"""

xontrib/coreutils.py

@@ -0,0 +1,30 @@
+"""Additional core utilites that are implemented in xonsh. The current list
+includes:
+
+* cat
+* echo
+* pwd
+* tee
+* tty
+* yes
+
+In many cases, these may have a lower performance overhead than the
+posix command line utility with the same name. This is because these
+tools avoid the need for a full subprocess call. Additionally, these
+tools are cross-platform.
+"""
+from xonsh.xoreutils.cat import cat
+from xonsh.xoreutils.echo import echo
+from xonsh.xoreutils.pwd import pwd
+from xonsh.xoreutils.tee import tee
+from xonsh.xoreutils.tty import tty
+from xonsh.xoreutils.yes import yes
+
+__all__ = ()
+
+aliases['cat'] = cat
+aliases['echo'] = echo
+aliases['pwd'] = pwd
+aliases['tee'] = tee
+aliases['tty'] = tty
+aliases['yes'] = yes

xontrib/mpl.py

@@ -1,11 +1,15 @@
-"""Matplotlib xontribution."""
+"""Matplotlib xontribution. This xontrib should be loaded before matplotlib
+is imported.
+"""
+
+from xonsh.tools import unthreadable
+from xonsh.lazyasd import lazyobject
 
-from xonsh.proc import foreground as foreground
 
 __all__ = ()
 
 
-@foreground
+@unthreadable
 def mpl(args, stdin=None):
     """Hooks to matplotlib"""
     from xontrib.mplhooks import show
@@ -13,3 +17,48 @@ def mpl(args, stdin=None):
 
 
 aliases['mpl'] = mpl
+
+
+@lazyobject
+def pylab_helpers():
+    try:
+        import matplotlib._pylab_helpers as m
+    except ImportError:
+        m = None
+    return m
+
+
+@events.on_import_post_exec_module
+def interactive_pyplot(module=None, **kwargs):
+    """This puts pyplot in interactive mode once it is imported."""
+    if module.__name__ != 'matplotlib.pyplot' or \
+       not __xonsh_env__.get('XONSH_INTERACTIVE'):
+        return
+    # Since we are in interactive mode, let's monkey-patch plt.show
+    # to try to never block.
+    module.ion()
+    module._INSTALL_FIG_OBSERVER = False
+    plt_show = module.show
+
+    def xonsh_show(*args, **kwargs):
+        """This is a monkey patched version of matplotlib.pyplot.show()
+        for xonsh's interactive mode. First it tries non-blocking mode
+        (block=False). If for some reason this fails, it will run show
+        in normal bloking mode (block=True).
+        """
+        kwargs.update(block=False)
+        rtn = plt_show(*args, **kwargs)
+        figmanager = pylab_helpers.Gcf.get_active()
+        if figmanager is not None:
+            # unblocked mode failed, try blocking.
+            kwargs.update(block=True)
+            rtn = plt_show(*args, **kwargs)
+        return rtn
+
+    module.show = xonsh_show
+
+    # register figure drawer
+    @events.on_postcommand
+    def redraw_mpl_figure(**kwargs):
+        """Redraws the current matplotlib figure after each command."""
+        pylab_helpers.Gcf.draw_all()