Environment variable registration, deregistration (addresses #3227) (#3377)

* First attempt at register/deregister machinery for envvars

* Added detailed docstring, simplified ensurers

Also added some type checking that became clear from docstring writing.

* Changes in response to @scopatz review

Simplified kwarg names.

* defaultval -> default

* Created new Var namedtuple, as well as DEFAULT_VARS

We should now delete DEFAULT_ENSURERS, DEFAULT_VALUES, DEFAULT_DOCS, and
refactor Env to use the new single namedtuple and the DEFAULT_VARS dict

* Removed DEFAULT_ENSURERS, DEFAULT_VALUES, DEFAULT_DOCS

Now need to edit Env to use new DEFAULT_VARS, Var namedtuple

* Finished updating Env object to use new combined Var

Also made corresponding changes elsewhere ensurer was used

* Working on test failures

* More fixes in light of test failures

* Set default values for Var in register.

There's a bit of duplication here, but makes for a cleaner function.

* Black reformatting on environ.py

* Removed history replay

* Added register tests

* Added addtional deregistration test

* Removed all replay references, in docs too

* Added news item for env-reg-dereg

* trigger rebuild

* doc fix

* more doc fixes

* again

* attr names

* reorder imports

* fix flake error

Co-authored-by: Anthony Scopatz <scopatz@gmail.com>
This commit is contained in:
David Dotson 2020-08-05 07:39:11 -07:00 committed by GitHub
parent 784c5286ef
commit a8d4a57f01
Failed to generate hash of commit
16 changed files with 1028 additions and 826 deletions

View file

@ -92,14 +92,6 @@ for more information all the history command and all of its sub-commands.
.. command-help:: xonsh.history.main.history_main
``replay``
=====================
Replays a xonsh history file. See `the replay section of the history tutorial
<tutorial_hist.html#replay-action>`_ for more information about this command.
.. command-help:: xonsh.replay.replay_main
``timeit``
===============
Runs timing study on arguments. Similar to IPython's ``%timeit`` magic.

View file

@ -49,7 +49,6 @@ For those of you who want the gritty details.
ptk_shell/completer
ptk_shell/key_bindings
pretty
replay
diff_history
xoreutils/index

View file

@ -1,10 +0,0 @@
.. _xonsh_replay:
******************************************************
Replay History (``xonsh.replay``)
******************************************************
.. automodule:: xonsh.replay
:members:
:undoc-members:
:inherited-members:

View file

@ -16,7 +16,7 @@ import importlib
os.environ["XONSH_DEBUG"] = "1"
from xonsh import __version__ as XONSH_VERSION
from xonsh.environ import DEFAULT_DOCS, Env
from xonsh.environ import DEFAULT_VARS, Env
from xonsh.xontribs import xontrib_metadata
from xonsh import main
from xonsh.commands_cache import CommandsCache
@ -283,12 +283,13 @@ runthis_server = "https://runthis.xonsh.org:80"
def make_envvars():
env = Env()
vars = sorted(DEFAULT_DOCS.keys())
vars = sorted(DEFAULT_VARS.keys(), key=lambda x: getattr(x, "pattern", x))
s = ".. list-table::\n" " :header-rows: 0\n\n"
table = []
ncol = 3
row = " {0} - :ref:`${1} <{2}>`"
for i, var in enumerate(vars):
for i, varname in enumerate(vars):
var = getattr(varname, "pattern", varname)
star = "*" if i % ncol == 0 else " "
table.append(row.format(star, var, var.lower()))
table.extend([" -"] * ((ncol - len(vars) % ncol) % ncol))
@ -304,18 +305,19 @@ def make_envvars():
"**store_as_str:** {store_as_str}\n\n"
"-------\n\n"
)
for var in vars:
for varname in vars:
var = getattr(varname, "pattern", varname)
title = "$" + var
under = "." * len(title)
vd = env.get_docs(var)
vd = env.get_docs(varname)
s += sec.format(
low=var.lower(),
title=title,
under=under,
docstr=vd.docstr,
configurable=vd.configurable,
default=vd.default,
store_as_str=vd.store_as_str,
docstr=vd.doc,
configurable=vd.doc_configurable,
default=vd.doc_default,
store_as_str=vd.doc_store_as_str,
)
s = s[:-9]
fname = os.path.join(os.path.dirname(__file__), "envvarsbody")

View file

@ -58,18 +58,8 @@ So the reasons for having rich history are debugging and reproducibility. Xonsh
guess-work out of the past. There is even the ability to store all of stdout, though this
is turned off by default.
If history was just a static file, it would be more like a server log than a traditional
history file. However, xonsh also has the ability to ``replay`` a history file.
history file.
Replaying history allows previous sessions to act as scripts in a new or the same environment.
Replaying will create a new, separate history session and file. The two histories - even though
they contain the same inputs - are then able to be diff'ed. Diff'ing can be done through
xonsh custom history diff'ing tool, which can help pinpoint differences stemming from the
environment as well as the input/output. This cycle of do-replay-diff is more meaningful than
a traditional, "What did I/it/the Universe just do?!" approach.
Of course, nothing has ever stopped anyone from pulling Unix tools like ``env``, ``script``,
``diff``, and others together to deliver the same kind of capability. However, in practice,
no one does this. With xonsh, rich and useful history come batteries included.
``history`` command
====================
@ -192,79 +182,6 @@ series of lines. However, it can also return a JSON formatted string.
"filename": "/home/scopatz/.local/share/xonsh/xonsh-ace97177-f8dd-4a8d-8a91-a98ffd0b3d17.json",
"length": 7, "buffersize": 100, "bufferlength": 7}
``replay`` action
==================
The ``replay`` action allows for history files to be rerun, as scripts or in an existing xonsh
session.
First, the original ``'replay'`` environment is loaded and will be merged with the current ``'native'``
environment. How the environments are merged or not merged can be set at replay time. The default is for
the current native environment to take precedence. Next, each input in the environment is executed in order.
Lastly, the information of the replayed history file is printed.
Let's walk through an example. To begin with, open up xonsh and run some simple commands, as follows.
Call this the ``orig`` session.
**orig history**
.. code-block:: xonshcon
>>> mkdir -p temp/
>>> cd temp
>>> import random
>>> touch @(random.randint(0, 18))
>>> ls
2
>>> history file
/home/scopatz/.local/share/xonsh/xonsh-4bc4ecd6-3eba-4f3a-b396-a229ba2b4810.json
>>> exit
We can now replay this by passing the filename into the replay command or the replay action
of the history command. This action has a few different options, but one of them is that
we can select a different target output file with the ``-o`` or ``--target`` option.
For example, in a new session, we could run:
**new history**
.. code-block:: xonshcon
>>> history replay -o ~/new.json ~/.local/share/xonsh/xonsh-4bc4ecd6-3eba-4f3a-b396-a229ba2b4810.json
2 10
/home/scopatz/new.json
----------------------------------------------------------------
Just replayed history, new history the has following information
----------------------------------------------------------------
sessionid: 35712b6f-4b15-4ef9-8ce3-b4c781601bc2
filename: /home/scopatz/new.json
length: 7
buffersize: 100
bufferlength: 0
As you can see, a new history was created and another random file was added to the file system.
If we want instead to replay history in its own session, we can always use the ``-c`` option on
xonsh itself to execute the replay command.
**next history**
.. code-block:: xonshcon
>>> xonsh -c "replay -o ~/next.json ~/new.json"
2 7 10
/home/scopatz/next.json
----------------------------------------------------------------
Just replayed history, new history has the following information
----------------------------------------------------------------
sessionid: 70d7186e-3eb9-4b1c-8f82-45bb8a1b7967
filename: /home/scopatz/next.json
length: 7
buffersize: 100
bufferlength: 0
Currently history does not handle alias storage and reloading, but such a feature may be coming in
the future.
``diff`` action
===============
@ -277,7 +194,7 @@ is to be meaningful. However, they don't need to be exactly the same.
The diff action has one major option, ``-v`` or ``--verbose``. This basically says whether the
diff should go into as much detail as possible or only pick out the relevant pieces. Diffing
the new and next examples from the replay action, we see the diff looks like:
the new and next examples, we see the diff looks like:
.. code-block:: xonshcon
@ -322,7 +239,7 @@ As can be seen, the diff has three sections.
1. **The header** describes the meta-information about the histories, such as
their file names, sessionids, and time stamps.
2. **The environment** section describes the differences in the environment
when the histories were started or replayed.
when the histories were started.
3. **The commands** list this differences in the command themselves.
For the commands, the input sequences are diff'd first, prior to the outputs
@ -443,8 +360,7 @@ of hocus pocus before you get to anything real.
Xonsh has implemented a generic indexing system (sizes, offsets, etc)for JSON files that lives
inside of the file that it indexes. This is known as ``LazyJSON`` because it allows us to
only read in the parts of a file that we need. For example, for replaying we only need to
grab the input fields and so that helps us on I/O. For garbage collecting based on the number
only read in the parts of a file that we need. For garbage collecting based on the number
of commands, we can get this information from the index and don't need to read in any of the
original data.
@ -474,8 +390,8 @@ Sqlite History Backend
Xonsh has a second built-in history backend powered by sqlite (other than
the JSON version mentioned all above in this tutorial). It shares the same
functionality as the JSON version in most ways, except it currently doesn't
support ``history diff`` and ``history replay`` actions and does not store
the output of commands, as the json-backend does. E.g.
support the ``history diff`` action and does not store the output of commands,
as the json-backend does. E.g.
`__xonsh__.history[-1].out` will always be `None`.
The Sqlite history backend can provide a speed advantage in loading history

24
news/env-reg-dereg.rst Normal file
View file

@ -0,0 +1,24 @@
**Added:**
* <news item>
**Changed:**
* Added ability to register, deregister environment variables;
centralized environment default variables
**Deprecated:**
* <news item>
**Removed:**
* Removed history replay
**Fixed:**
* <news item>
**Security:**
* <news item>

View file

@ -11,10 +11,8 @@ import pytest
from xonsh.commands_cache import CommandsCache
from xonsh.environ import (
Env,
Ensurer,
locate_binary,
DEFAULT_ENSURERS,
DEFAULT_VALUES,
DEFAULT_VARS,
default_env,
make_args_env,
LsColors,
@ -67,7 +65,7 @@ def test_env_detype_mutable_access_clear(path1, path2):
def test_env_detype_no_dict():
env = Env(YO={"hey": 42})
env.set_ensurer("YO", Ensurer(always_true, None, None))
env.register("YO", validate=always_true, convert=None, detype=None)
det = env.detype()
assert "YO" not in det
@ -256,13 +254,6 @@ def test_events_on_envvar_called_in_right_order(xonsh_builtins):
assert share == ["change"]
def test_int_bool_envvars_have_ensurers():
bool_ints = [type(envvar) in [bool, int] for envvar in DEFAULT_VALUES.values()]
key_mask = set(itertools.compress(DEFAULT_VALUES.keys(), bool_ints))
ensurer_keys = set(DEFAULT_ENSURERS.keys())
assert len(key_mask.intersection(ensurer_keys)) == len(key_mask)
def test_no_lines_columns():
os.environ["LINES"] = "spam"
os.environ["COLUMNS"] = "eggs"
@ -298,7 +289,7 @@ def test_delitem():
def test_delitem_default():
env = Env()
a_key, a_value = next(
(k, v) for (k, v) in env._defaults.items() if isinstance(v, str)
(k, v.default) for (k, v) in env._vars.items() if isinstance(v.default, str)
)
del env[a_key]
assert env[a_key] == a_value
@ -348,3 +339,119 @@ def test_lscolors_events(key_in, old_in, new_in, test, xonsh_builtins):
assert not event_fired, "No event if value doesn't change"
else:
assert event_fired
def test_register_custom_var_generic():
"""Test that a registered envvar without any type is treated
permissively.
"""
env = Env()
assert "MY_SPECIAL_VAR" not in env
env.register("MY_SPECIAL_VAR")
assert "MY_SPECIAL_VAR" in env
env["MY_SPECIAL_VAR"] = 32
assert env["MY_SPECIAL_VAR"] == 32
env["MY_SPECIAL_VAR"] = True
assert env["MY_SPECIAL_VAR"] == True
def test_register_custom_var_int():
env = Env()
env.register("MY_SPECIAL_VAR", type='int')
env["MY_SPECIAL_VAR"] = "32"
assert env["MY_SPECIAL_VAR"] == 32
with pytest.raises(ValueError):
env["MY_SPECIAL_VAR"] = "wakka"
def test_register_custom_var_float():
env = Env()
env.register("MY_SPECIAL_VAR", type='float')
env["MY_SPECIAL_VAR"] = "27"
assert env["MY_SPECIAL_VAR"] == 27.0
with pytest.raises(ValueError):
env["MY_SPECIAL_VAR"] = "wakka"
@pytest.mark.parametrize("val,converted",
[
(True, True),
(32, True),
(0, False),
(27.0, True),
(None, False),
("lol", True),
("false", False),
("no", False),
])
def test_register_custom_var_bool(val, converted):
env = Env()
env.register("MY_SPECIAL_VAR", type='bool')
env["MY_SPECIAL_VAR"] = val
assert env["MY_SPECIAL_VAR"] == converted
@pytest.mark.parametrize("val,converted",
[
(32, "32"),
(0, "0"),
(27.0, "27.0"),
(None, "None"),
("lol", "lol"),
("false", "false"),
("no", "no"),
])
def test_register_custom_var_str(val, converted):
env = Env()
env.register("MY_SPECIAL_VAR", type='str')
env["MY_SPECIAL_VAR"] = val
assert env["MY_SPECIAL_VAR"] == converted
def test_register_custom_var_path():
env = Env()
env.register("MY_SPECIAL_VAR", type='path')
paths = ["/home/wakka", "/home/wakka/bin"]
env["MY_SPECIAL_VAR"] = paths
assert hasattr(env['MY_SPECIAL_VAR'], 'paths')
assert env["MY_SPECIAL_VAR"].paths == paths
with pytest.raises(TypeError):
env["MY_SPECIAL_VAR"] = 32
def test_deregister_custom_var():
env = Env()
env.register("MY_SPECIAL_VAR", type='path')
env.deregister("MY_SPECIAL_VAR")
assert "MY_SPECIAL_VAR" not in env
env.register("MY_SPECIAL_VAR", type='path')
paths = ["/home/wakka", "/home/wakka/bin"]
env["MY_SPECIAL_VAR"] = paths
env.deregister("MY_SPECIAL_VAR")
# deregistering a variable that has a value set doesn't
# remove it from env;
# the existing variable also maintains its type validation, conversion
assert "MY_SPECIAL_VAR" in env
with pytest.raises(TypeError):
env["MY_SPECIAL_VAR"] = 32
# removing, then re-adding the variable without registering
# gives it only default permissive validation, conversion
del env["MY_SPECIAL_VAR"]
env["MY_SPECIAL_VAR"] = 32

View file

@ -1,47 +0,0 @@
# -*- coding: utf-8 -*-
"""Tests the xonsh replay functionality."""
import os
import builtins
import pytest
from xonsh.shell import Shell
from xonsh.execer import Execer
from xonsh.replay import Replayer
from tools import skip_if_on_darwin
HISTDIR = os.path.join(os.path.dirname(__file__), "histories")
@pytest.fixture(scope="module", autouse=True)
def ctx():
"""Create a global Shell instance to use in all the test."""
ctx = {"PATH": []}
execer = Execer(xonsh_ctx=ctx)
builtins.__xonsh__.shell = Shell(execer=execer, ctx=ctx, shell_type="none")
yield
builtins.__xonsh__.shell = None
@skip_if_on_darwin
def test_echo():
histfile = os.path.join(HISTDIR, "echo.json")
hist = Replayer(histfile).replay()
assert len(hist) == 2
@skip_if_on_darwin
def test_reecho():
histfile = os.path.join(HISTDIR, "echo.json")
hist = Replayer(histfile).replay()
assert len(hist) == 2
@skip_if_on_darwin
def test_simple_python():
histfile = os.path.join(HISTDIR, "simple-python.json")
hist = Replayer(histfile).replay()
assert len(hist) == 4
assert hist.inps[0].strip() == "print('The Turtles')"

View file

@ -80,16 +80,14 @@ else:
_sys.modules["xonsh.environ"] = __amalgam__
tracer = __amalgam__
_sys.modules["xonsh.tracer"] = __amalgam__
readline_shell = __amalgam__
_sys.modules["xonsh.readline_shell"] = __amalgam__
replay = __amalgam__
_sys.modules["xonsh.replay"] = __amalgam__
aliases = __amalgam__
_sys.modules["xonsh.aliases"] = __amalgam__
dumb_shell = __amalgam__
_sys.modules["xonsh.dumb_shell"] = __amalgam__
readline_shell = __amalgam__
_sys.modules["xonsh.readline_shell"] = __amalgam__
built_ins = __amalgam__
_sys.modules["xonsh.built_ins"] = __amalgam__
dumb_shell = __amalgam__
_sys.modules["xonsh.dumb_shell"] = __amalgam__
execer = __amalgam__
_sys.modules["xonsh.execer"] = __amalgam__
imphooks = __amalgam__

View file

@ -34,7 +34,6 @@ from xonsh.tools import (
unthreadable,
print_color,
)
from xonsh.replay import replay_main
from xonsh.timings import timeit_alias
from xonsh.xontribs import xontribs_main
from xonsh.ast import isexpression
@ -781,7 +780,6 @@ def make_default_aliases():
"source-cmd": source_cmd,
"source-foreign": source_foreign,
"history": xhm.history_main,
"replay": replay_main,
"trace": trace,
"timeit": timeit_alias,
"umask": umask,

File diff suppressed because it is too large Load diff

View file

@ -333,7 +333,7 @@ class JsonCommandField(cabc.Sequence):
class JsonHistory(History):
"""Xonsh history backend implemented with JSON files.
JsonHistory implements two extra actions: ``diff``, and ``replay``.
JsonHistory implements an extra action: ``diff``
"""
def __init__(self, filename=None, sessionid=None, buffersize=100, gc=True, **meta):

View file

@ -351,12 +351,6 @@ def _xh_create_parser():
diff = subp.add_parser("diff", help="diff two xonsh history files")
xdh.dh_create_parser(p=diff)
import xonsh.replay as xrp
replay = subp.add_parser("replay", help="replay a xonsh history file")
xrp.replay_create_parser(p=replay)
_XH_MAIN_ACTIONS.add("replay")
# 'flush' subcommand
subp.add_parser("flush", help="flush the current history to disk")
@ -419,11 +413,6 @@ def history_main(
elif ns.action == "diff":
if isinstance(hist, JsonHistory):
xdh.dh_main_action(ns)
elif ns.action == "replay":
if isinstance(hist, JsonHistory):
import xonsh.replay as xrp
xrp.replay_main_action(hist, ns, stdout=stdout, stderr=stderr)
elif ns.action == "flush":
hf = hist.flush()
if isinstance(hf, threading.Thread):

View file

@ -1,145 +0,0 @@
# -*- coding: utf-8 -*-
"""Tools to replay xonsh history files."""
import json
import time
import builtins
import collections.abc as cabc
from xonsh.tools import swap
from xonsh.lazyjson import LazyJSON
from xonsh.environ import Env
import xonsh.history.main as xhm
DEFAULT_MERGE_ENVS = ("replay", "native")
class Replayer(object):
"""Replays a xonsh history file."""
def __init__(self, f, reopen=True):
"""
Parameters
----------
f : file handle or str
Path to xonsh history file.
reopen : bool, optional
Whether new file handle should be opened for each load, passed directly into
LazyJSON class.
"""
self._lj = LazyJSON(f, reopen=reopen)
def __del__(self):
self._lj.close()
def replay(self, merge_envs=DEFAULT_MERGE_ENVS, target=None):
"""Replays the history specified, returns the history object where the code
was executed.
Parameters
----------
merge_env : tuple of str or Mappings, optional
Describes how to merge the environments, in order of increasing precedence.
Available strings are 'replay' and 'native'. The 'replay' env comes from the
history file that we are replaying. The 'native' env comes from what this
instance of xonsh was started up with. Instead of a string, a dict or other
mapping may be passed in as well. Defaults to ('replay', 'native').
target : str, optional
Path to new history file.
"""
shell = builtins.__xonsh__.shell
re_env = self._lj["env"].load()
new_env = self._merge_envs(merge_envs, re_env)
new_hist = xhm.construct_history(
env=new_env.detype(),
locked=True,
ts=[time.time(), None],
gc=False,
filename=target,
)
with swap(builtins.__xonsh__, "env", new_env), swap(
builtins.__xonsh__, "history", new_hist
):
for cmd in self._lj["cmds"]:
inp = cmd["inp"]
shell.default(inp)
if builtins.__xonsh__.exit: # prevent premature exit
builtins.__xonsh__.exit = False
new_hist.flush(at_exit=True)
return new_hist
def _merge_envs(self, merge_envs, re_env):
new_env = {}
for e in merge_envs:
if e == "replay":
new_env.update(re_env)
elif e == "native":
new_env.update(builtins.__xonsh__.env)
elif isinstance(e, cabc.Mapping):
new_env.update(e)
else:
raise TypeError("Type of env not understood: {0!r}".format(e))
new_env = Env(**new_env)
return new_env
_REPLAY_PARSER = None
def replay_create_parser(p=None):
global _REPLAY_PARSER
p_was_none = p is None
if _REPLAY_PARSER is not None and p_was_none:
return _REPLAY_PARSER
if p_was_none:
from argparse import ArgumentParser
p = ArgumentParser("replay", description="replays a xonsh history file")
p.add_argument(
"--merge-envs",
dest="merge_envs",
default=DEFAULT_MERGE_ENVS,
nargs="+",
help="Describes how to merge the environments, in order of "
"increasing precedence. Available strings are 'replay' and "
"'native'. The 'replay' env comes from the history file that we "
"are replaying. The 'native' env comes from what this instance "
"of xonsh was started up with. One or more of these options may "
"be passed in. Defaults to '--merge-envs replay native'.",
)
p.add_argument(
"--json",
dest="json",
default=False,
action="store_true",
help="print history info in JSON format",
)
p.add_argument(
"-o", "--target", dest="target", default=None, help="path to new history file"
)
p.add_argument("path", help="path to replay history file")
if p_was_none:
_REPLAY_PARSER = p
return p
def replay_main_action(h, ns, stdout=None, stderr=None):
replayer = Replayer(ns.path)
hist = replayer.replay(merge_envs=ns.merge_envs, target=ns.target)
print("----------------------------------------------------------------")
print("Just replayed history, new history has the following information")
print("----------------------------------------------------------------")
data = hist.info()
if ns.json:
s = json.dumps(data)
print(s, file=stdout)
else:
lines = ["{0}: {1}".format(k, v) for k, v in data.items()]
print("\n".join(lines), file=stdout)
def replay_main(args, stdin=None):
"""Acts as main function for replaying a xonsh history file."""
parser = replay_create_parser()
ns = parser.parse_args(args)
replay_main_action(ns)

View file

@ -2125,9 +2125,9 @@ def expandvars(path):
for match in POSIX_ENVVAR_REGEX.finditer(path):
name = match.group("envvar")
if name in env:
ensurer = env.get_ensurer(name)
detyper = env.get_detyper(name)
val = env[name]
value = str(val) if ensurer.detype is None else ensurer.detype(val)
value = str(val) if detyper is None else detyper(val)
value = str(val) if value is None else value
path = POSIX_ENVVAR_REGEX.sub(value, path, count=1)
return path

View file

@ -184,8 +184,8 @@ def _dump_xonfig_foreign_shell(path, value):
def _dump_xonfig_env(path, value):
name = os.path.basename(path.rstrip("/"))
ensurer = builtins.__xonsh__.env.get_ensurer(name)
dval = str(value) if ensurer.detype is None else ensurer.detype(value)
detyper = builtins.__xonsh__.env.get_detyper(name)
dval = str(value) if detyper is None else detyper(value)
dval = str(value) if dval is None else dval
return "${name} = {val!r}".format(name=name, val=dval)
@ -314,11 +314,11 @@ def make_envvar(name):
docstr=_wrap_paragraphs(vd.docstr, width=69),
)
mnode = wiz.Message(message=msg)
ens = env.get_ensurer(name)
converter = env.get_converter(name)
path = "/env/" + name
pnode = wiz.StoreNonEmpty(
ENVVAR_PROMPT,
converter=ens.convert,
converter=converter,
show_conversion=True,
path=path,
retry=True,