### Motivation
There is annoying behavior when you run command in the loop and can't
interrupt e.g. [this
report](https://github.com/xonsh/xonsh/discussions/5371) and a bit
#5342. After diving into this I see the issue around return code.
### The essence
Basically ``p = subprocess.Popen()`` populates ``p.returncode`` after
``p.wait()``, ``p.poll()`` or ``p.communicate()``
([doc](https://docs.python.org/3/library/os.html#os.waitpid)). But if
you're using `os.waitpid()` BEFORE these functions you're capturing
return code from a signal subsystem and ``p.returncode`` will be ``0``
like success but it's not success. So after ``os.waitid`` call you need
to set return code manually ``p.returncode = -os.WTERMSIG(status)`` like
in Popen. Example:
```xsh
python # python interactive
import os, signal, subprocess as sp
p = sp.Popen(['sleep', '100'])
p.pid
# 123
p.wait()
# Ctrl+C or `kill -SIGINT 123` from another terminal
p.returncode
# -2
# BUT:
p = sp.Popen(['sleep', '100'])
p.pid
# 123
pid, status = os.waitpid(p.pid, os.WUNTRACED)
# status=2
# From another terminal:
kill -SIGINT 123
p.wait()
# 0
p.returncode
# 0
```
```xsh
from xonsh.tools import describe_waitpid_status
describe_waitpid_status(2)
# WIFEXITED - False - Return True if the process returning status exited via the exit() system call.
# WEXITSTATUS - 0 - Return the process return code from status.
# WIFSIGNALED - True - Return True if the process returning status was terminated by a signal.
# WTERMSIG - 2 - Return the signal that terminated the process that provided the status value.
# WIFSTOPPED - False - Return True if the process returning status was stopped.
# WSTOPSIG - 0 - Return the signal that stopped the process that provided the status value.
```
See also: [Helpful things for
knight](https://github.com/xonsh/xonsh/pull/5361#issuecomment-2078826181).
### Before
```xsh
$RAISE_SUBPROC_ERROR = True
sleep 100
# Ctrl+C
_.rtn
# 0 # It's wrong and RAISE_SUBPROC_ERROR ignored.
for i in range(5):
print(i)
sleep 5
# 0
# Ctrl+C # Can't interrupt
# 1
# 2
```
### After
```xsh
sleep 100
# Ctrl+C
_.rtn
# -2 # Like in default Popen
$RAISE_SUBPROC_ERROR = True
for i in range(5):
print(i)
sleep 5
# 0
# Ctrl+C
# subprocess.CalledProcessError
```
### Notes
* We need to refactor `xonsh.jobs`. It's pretty uncomfortable work with
module.
* The logic is blurry between Specs, Pipelines and Jobs. We need to
bring more clear functions.
* The `captured` variable looks like "just the way of capturing (stdout,
object)" but in fact it affects all logic very much. We need to create
table where we can see the logic difference for every type of capturing.
E.g. in `captured=stdout` mode we use `xonsh.jobs` to monitor the
command but in `captured=object` we avoid this and some logic from
`xonsh.jobs` applied in `stdout` mode but missed in `object` mode. We
need clear map.
## For community
⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍
comment**
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
### Motivation
* To have an ability to manage the output format added ``$XONSH_SUBPROC_OUTPUT_FORMAT`` to switch the way to return the output lines. Default ``stream_lines`` to return text. Alternative ``list_lines`` to return the list of lines. Also supported custom lambda function.
* Additionally the [proposal to change default behavior](https://github.com/xonsh/xonsh/pull/5377#discussion_r1587627131) for a single line case.
* Closes#3924 as soft solution.
### Before
```xsh
mkdir -p /tmp/tst && cd /tmp/tst && touch 1 2 3
$(ls)
# '1\n2\n3\n'
id $(whoami)
# id: ‘pc\n’: no such user: Invalid argument
du $(ls)
# du: cannot access '1'$'\n''2'$'\n''3'$'\n': No such file or directory
ls $(fzf)
# ls: cannot access 'FUNDING.yml'$'\n': No such file or directory
```
### After
```xsh
mkdir -p /tmp/tst && cd /tmp/tst && touch 1 2 3
$XONSH_SUBPROC_OUTPUT_FORMAT = 'list_lines'
$(ls)
# ['1', '2', '3']
[f for f in $(ls)]
# ['1', '2', '3']
id $(whoami)
# uid=501(user) gid=20(staff)
du $(ls)
# 0 1
# 0 2
# 0 3
ls $(fzf)
# FUNDING.yml
# etc
mkdir -p /tmp/@($(whoami))/dir
cat /etc/passwd | grep $(whoami)
```
### Notes
* It will be good to improve parser for cases like `mkdir -p /tmp/$(whoami)/dir`. PR is welcome!
* I named the default mode as `stream_lines` (instead of just `stream` or `raw`) because in fact we transform raw output into stream of lines and possibly reduce the length of output ([replacing `\r\n` to `\n`](c3a12b2a9c/xonsh/procs/pipelines.py (L380-L383))). May be some day we need to add raw "stream" output format.
* Now anybody can implement bash `IFS` behavior in [bashisms](https://github.com/xonsh/xontrib-bashisms).
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix parsing of redirect tokens
In the parser, wrap redirect tokens such as ">", "a>", and "2>1" in a
tuple (along with the name of the redirect file, if applicable) to
differentiate them from regular command line arguments.
This ensures that a command like ![echo ">"] is parsed correctly.
We now need 2 types of IO-redirect tokens (IOREDIRECT1 and IOREDIRECT2)
so that the parser knows whether or not an IO-redirect has a file
argument. For example, "a>" has type IOREDIRECT1, whereas "2>1" has
type IOREDIRECT2.
* update ioredir tests
* add tests
* add news
* fix multiple redirections error message
* add tests
* Fix incorrect IOREDIRECT tokens in Python mode
* add news
* Update lexer used in CompletionContextParser
The completion context lexer should start in subprocess mode so that
operators like ``2>`` are recognized as IOREDIRECT tokens.
This fixes the test case "command 2>/dev/nul{X}" for
test_completion_context.py::test_command.
`os.path.expanduser` relies on the value of `$HOME` set in `os.environ`.
This meant that a user who set `$HOME` would then not end up in that
directory with a no-argument call to `cd` unless they had set
`UPDATE_OS_ENVIRON`.
Now the value of `$HOME` is respected regardless of the setting in `UPDATE_OS_ENVIRON`
* fix: Correctly multi line aliases
Bash aliases can span multi lines for readablity or if they are to long
for one line.
Here we just remove added the backslash before further processing.
* doc: Add news item for fix/parse-multiline-aliases branch
---------
Co-authored-by: Wilfried Pollan <wilfried.pollan@moqodow.com>
* [pre-commit.ci] pre-commit autoupdate
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.0.278 → v0.0.280](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.278...v0.0.280)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: ruff linter errors
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Noorhteen Raja NJ <jnoortheen@gmail.com>
* refactor: project requires 3.9+
changed as per NEP-29
https://numpy.org/neps/nep-0029-deprecation_policy.html
* test: nose type tests are deprecated in pytest now
* fix: deprecation of ast.Str and ast.Bytes and .s attribute access
* fix: deprecation of ast.Num,ast.NameConstant,ast.Ellipsis
* refactor: upgrade code to be py39+ using ruff
the changes are auto-generated
* refactor: remove typing.Annotated compatibility code
* fix: temporarily disable having xonsh syntax inside f-strings
* test: skip failing tests
there is no workaround for this version. It might get solved in the
final release though
* refactor: make XonshSession.completer lazily populated
this speedsup the tests as cmd_cache would not be actively populated
when used in default_completers function
* refactor: make presence of walrus operator default
* Update and rename README to README.rst
* Create hello_world.py
* Update README.rst
* init
* remove bashisms
* Transfer abbrevs to xontrib-abbrevs
* Transfer free-cwd to xontrib-free-cwd
* news
* black
* Transfer fish-completer to xonsh/xontrib-fish-completer
* Transfer vox to xonsh/xontrib-vox
* Transfer pdb, xog to xonsh/xontrib-debug-tools
* remove hello_world
* fix tests
* black
* fix whitespaces
* fix readme
* Update python_virtual_environments.rst
* Update README.rst
* Update xontribs_transfer.rst
---------
Co-authored-by: a <1@1.1>
* Update tools.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* chdir
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* todo:
* refactor: remove usage of singleton in cmd-cache
* refactor: pass aliases/env to commands_cache on creation
reduce the usage of singletons
* fix: setting up mockups for tests
* feat: add $XONSH_CACHE_DIR
* refactor: use cache path to stire cmds-cache
* chore: todo
* feat: efficient cache commands per path and modified time
* refactor: move aliases to commands cache
now XonshSession is not needed to setup cmds-cache
* refactor: update tests
* fix: type annotation
* refactor: init aliases at commands-cache
* test: update failing tests
* fix: handle paths mtime changing
* docs: add news item
* refactor: remove $COMMANDS_CACHE_SIZE_WARNING
* fix: loading on windows fails because of setup sequence
* fix: failing tests
* Add XONSH_HISTORY_IGNORE_REGEX support
* Add changelog text
* Add cleanup notes for XONSH_HISTORY_IGNORE_REGEX
* Rename should_append to is_ignored
* Change is_ignored to a History method
* Remove unused re imports
* Docstring formatting cleanup, Black compliance
* Move XONSH_HISTORY_IGNORE_REGEX to History class
* Clarify with parenthesis
* Formatting compliance
* Add validate_ignore_regex check
* Remove validation comment
* Remove XONSH_HISTORY_IGNORE_REGEX test comments
* Add __init__ chaining to DummyHistory
* Add broken test for XONSH_IGNORE_REGEX
* Remove extra test variants
* Fix unit test environment variable name
* Add test for bad regex error output
* Remove unneeded __init__ chaining
* Add test_is_ignore
* Add is_regex and associated tests
* Add doc for XONSH_HISTORY_IGNORE_REGEX
* Change validate_ignore_regex to use is_regex
* Compile regex once and use a cached property
* Let cached_property handle saving regex
* Update terminal title before executing subprocess
This eliminates the need to use `pause_call_resume()`, which causes issues
when it sends SIGSTOP to a process that is in the middle of a syscall
(for example, the `swaymsg` command fails when it gets stopped during
a `recv` syscall).
Additionally, this approach is more flexible, as it allows captured
commands and callable aliases to also be updated as expected in the
terminal's title.
* Clear prompt cache in readline shell
* Add type annotation
* Use reset_key() to only reset current_job
* Convert into PromptField class
* Support PromptField with value of None
Allows {current_job:{} | } to work as expected when current_job is a
PromptField.
* Add tests
* Fixing bash completion bug when prefix started with '>', '<', or ':'
* Fixing code formatting
* More code formatting fixes
* Fixing test
* Refactor to fix bug by improving completion parser
* Supporting '<' and '>>' in completion parser
* Support for all IO redirect tokens
* Make job control task queue thread-local
This allows callable aliases to maintain a separate task queue from the
main thread.
* Make job control dictionary thread-local
Use XSH.all_jobs for the main thread and a separate dictionary for other
threads. This allows callable aliases to keep track of their jobs
separate from the main thread.
* Implement use_main_jobs() context manager
This allows threaded commands like jobs, disown, and bg to handle the
main thread's job control
* Run commands in the same process group if in a thread
* Fix tests that use jobs and tasks
* Add tests for subprocess call inside alias
* Add news
* Remove type declarations for _jobs_thread_local
Fixes the mypy error: "Type cannot be declared in assignment to non-self attribute"
Co-authored-by: Noorhteen Raja NJ <jnoortheen@gmail.com>
* Update test_pip_completer.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: add support for f-glob strings
Move xonsh_pathsearch() into the BaseParser class because it needs to use self._set_error()
Parametrize 6 backtick tests in test_parser.py into test_backtick() (and add cases for f-glob strings)
* add news
* docs: update tutorial
* fix news file
