Merge this after https://github.com/xonsh/xonsh/pull/5437
## For community
⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍
comment**
---------
Co-authored-by: a <1@1.1>
Reading stop signals from the process and update the process state.
### The issue
Technically. In a couple of places that critical for processing signals
we have `os.waitpid()`. The function behavior is pretty unobvious and
one of things is processing return code after catching the signal. We
had no good signal processing around this and this PR fixes this. See
also `proc_untraced_waitpid` function description.
From user perspective. For example we have process that is waiting for
user input from terminal e.g. `python -c "input()"` or `fzf`. If this
process will be in captured pipeline e.g. `!(echo 1 | fzf | head)` it
will be suspended by OS and the pipeline will be in the endless loop
with future crashing and corrupting std at the end. This PR fixes this.
### The solution
Technically. The key function is `proc_untraced_waitpid` - it catches
the stop signals and updates the process state.
From user perspective. First of all we expect that users will use
captured object `!()` only for capturable processes. Because of it our
goal here is to just make the behavior in this case stable.
In this PR we detect that process in the pipeline is suspended and we
need to finish the command pipeline carefully:
* Show the message about suspended process.
* Keep suspended process in `jobs`. The same behavior we can see in
bash. This is good because we don't know what process suspended and why.
May be experienced user will want to continue it manually.
* Finish the CommandPipeline with returncode=None and suspended=True.
### Before
```xsh
!(fzf) # or !(python -c "input()")
# Hanging / Exceptions / OSError / No way to end the command.
# After exception:
$(echo 1)
# OSError / IO error
```
### After
```xsh
!(fzf) # or `!(ls | fzf | head)` or `!(python -c "input()")`
# Process ['fzf'] with pid 60000 suspended with signal 22 SIGTTOU and stay in `jobs`.
# This happends when process start waiting for input but there is no terminal attached in captured mode.
# CommandPipeline(returncode=None, suspended=True, ...)
$(echo 1)
# Success.
```
Closes#4752#4577
### Notes
* There is pretty edge case situation when the process was terminated so
fast that we can't catch pid alive and check signal
([src](67d672783d/xonsh/jobs.py (L71-L80))).
I leave it as is for now.
### Mentions
#2159
## For community
⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍
comment**
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Gil Forsyth <gforsyth@users.noreply.github.com>
### Motivation
Development tools like PyCharm send SIGINT before SIGKILL in case of
reload app. Saving history on SIGINT is the last chance to save history
in this case.
I understand that history will be flushed on Ctrl+C (SIGINT) but it
looks like no bad consequences of this. In addition history saving will
be more stable.
### Before
```xsh
# Development in PyCharm
xonsh --no-rc
echo cmd
# Ctrl+RR of Ctrl+DD to restart.
# Command history lost.
```
### After
```xsh
# Development in PyCharm
xonsh --no-rc
echo cmd
# Ctrl+R Ctrl+R for restart
# Command history is here.
```
Closes#2657
### Env
Use `$XONSH_HISTORY_SIGINT_FLUSH=False` to switch it off.
## For community
⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍
comment**
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
### Motivation
There is no way to access to the CommandPipeline of executed command in
uncaptured mode. This causes:
* No chance to get return code in one universal way.
* Barrier to trace and inspection.
### Before
```xsh
$(ls nofile)
# No way to access to the CommandPipeline of executed command.
```
### After
```xsh
$(ls nofile)
__xonsh__.last.rtn # In interactive or not interactive.
# 2
```
```xsh
# Sugar sir:
last = type('LastCP', (object,), {'__getattr__':lambda self, name: getattr(__xonsh__.last, name) })()
ls nofile
last.rtn
# 2
```
JFYI #5342
## For community
⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍
comment**
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
### Motivation
Closes#4636
In fact`Env` manages two lists: explicitly set values and Xettings. When
we operate with env we have no detyping for Xettings. The `detype_all`
function doint detyping for all possible env variables.
### Before
```xsh
# No way to get all detyped variables e.g.
'PROMPT' in __xonsh__.env.detype()
# False
```
### After
```xsh
'PROMPT' in __xonsh__.env.detype_all()
# True
```
## For community
⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍
comment**
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
### Before
```xsh
echo 1
echo 2
# <<Crash>>
# No json history
```
### After
Closes#2657
```xsh
echo 1
echo 2
# <<Crash>>
# In json history
# echo 1
# echo 2
```
## For community
⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍
comment**
---------
Co-authored-by: a <1@1.1>
When cls.f is partial it's better to show full specification.
## For community
⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍
comment**
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Because we downgraded ptk in https://github.com/xonsh/xonsh/pull/5403
### Before
```xsh
xonsh --no-rc
# /Users/pc/.local/mamba-envs/lib/python3.12/site-packages/prompt_toolkit/application/application.py:961: DeprecationWarning: There is no current event loop
# loop = asyncio.get_event_loop()
#@
```
### After
```xsh
xonsh --no-rc
#@
```
## For community
⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍
comment**
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
### Before
Case 1: Handler catches the exit signal and do not pass it forward. As
result xonsh could be suspended by OS or crash. Also exit code is wrong.
See repeatable examples with SIGHUP in
https://github.com/xonsh/xonsh/issues/5381#issuecomment-2097961804.
Case 2: From bash/zsh as login shell run xonsh. Then send quit signal to
xonsh. The terminal state will be broken: disabled SIGINT, mouse pointer
produces mouse state codes.
### After
Case 1: Xonsh exit normally with right exit code. Fixed#5381#5304#5371.
Case 2: After exiting with right exit code the state of the terminal is
good.
## For community
⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍
comment**
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This downgrade continues previous (#5288). Found issue in prompt-toolkit
3.0.40 (#5393).
JFYI #5241
## For community
⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍
comment**
Co-authored-by: a <1@1.1>
### Before
Case 1:
```xsh
xonsh --no-rc --no-env -DCOLOR_OUTPUT 0 # space between -DVAR and VALUE
```
```xsh
Traceback (most recent call last):
File "/Users/pc/.local/xonsh-env/lib/python3.12/site-packages/xonsh/main.py", line 478, in main
args = premain(argv)
^^^^^^^^^^^^^
File "/Users/pc/.local/xonsh-env/lib/python3.12/site-packages/xonsh/main.py", line 420, in premain
env.update([x.split("=", 1) for x in args.defines])
File "<frozen _collections_abc>", line 987, in update
ValueError: not enough values to unpack (expected 2, got 1)
Xonsh encountered an issue during launch
```
Case 2:
```xsh
xonsh --no-rc --no-env -DXONSH_HISTORY_BACKEND=dummy
history info
# json # EXPECTED: dummy
```
### After
Case 1:
```xsh
xonsh --no-rc --no-env -DCOLOR_OUTPUT 0
# Wrong format for -DCOLOR_OUTPUT argument. Use -DVAR=VAL form.
```
Case 2:
```xsh
xonsh --no-rc --no-env -DXONSH_HISTORY_BACKEND=dummy
history info
# dummy
```
Closes#5395
## For community
⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍
comment**
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Fixed#5387
### Before
```xsh
$XONSH_SHOW_TRACEBACK = False
$RAISE_SUBPROC_ERROR = True
ls nofile
# ls: nofile: No such file or directory
# Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
#
```
### After
```xsh
$RAISE_SUBPROC_ERROR = False
$XONSH_SHOW_TRACEBACK = False
ls nofile
# ls: nofile: No such file or directory
$RAISE_SUBPROC_ERROR = True
$XONSH_SHOW_TRACEBACK = False
ls nofile
# ls: nofile: No such file or directory
# subprocess.CalledProcessError: Command '['ls', 'nofile']' returned non-zero exit status 1.
$RAISE_SUBPROC_ERROR = True
$XONSH_SHOW_TRACEBACK = True
ls nofile
# ls: nofile: No such file or directory
# Traceback (most recent call last):
# ...
# subprocess.CalledProcessError: Command '['ls', 'nofile']' returned non-zero exit status 1.
$RAISE_SUBPROC_ERROR = False
$XONSH_SHOW_TRACEBACK = True
ls nofile
# ls: nofile: No such file or directory
```
## For community
⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍
comment**
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Reverting xonsh/xonsh#5389 with update by comments
---------
Co-authored-by: Gil Forsyth <gforsyth@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
### Motivation
It's very annoying to search env variable exactly by lprefix. The better
is to search by substring and sort results by the position of substring
and then alphabetically.
Closes#5386
### Before
```xsh
$TRA<Tab>
# nothing
```
### After
```xsh
$TRA<Tab>
# 'XONSH_TRACE_COMPLETIONS',
# 'XONSH_TRACE_SUBPROC',
# 'XONSH_TRACE_SUBPROC_FUNC',
# 'XONSH_TRACEBACK_LOGFILE',
# 'XONSH_SHOW_TRACEBACK',
# 'VC_GIT_INCLUDE_UNTRACKED'
```
## For community
⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍
comment**
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
### Motivation
There is annoying behavior when you run command in the loop and can't
interrupt e.g. [this
report](https://github.com/xonsh/xonsh/discussions/5371) and a bit
#5342. After diving into this I see the issue around return code.
### The essence
Basically ``p = subprocess.Popen()`` populates ``p.returncode`` after
``p.wait()``, ``p.poll()`` or ``p.communicate()``
([doc](https://docs.python.org/3/library/os.html#os.waitpid)). But if
you're using `os.waitpid()` BEFORE these functions you're capturing
return code from a signal subsystem and ``p.returncode`` will be ``0``
like success but it's not success. So after ``os.waitid`` call you need
to set return code manually ``p.returncode = -os.WTERMSIG(status)`` like
in Popen. Example:
```xsh
python # python interactive
import os, signal, subprocess as sp
p = sp.Popen(['sleep', '100'])
p.pid
# 123
p.wait()
# Ctrl+C or `kill -SIGINT 123` from another terminal
p.returncode
# -2
# BUT:
p = sp.Popen(['sleep', '100'])
p.pid
# 123
pid, status = os.waitpid(p.pid, os.WUNTRACED)
# status=2
# From another terminal:
kill -SIGINT 123
p.wait()
# 0
p.returncode
# 0
```
```xsh
from xonsh.tools import describe_waitpid_status
describe_waitpid_status(2)
# WIFEXITED - False - Return True if the process returning status exited via the exit() system call.
# WEXITSTATUS - 0 - Return the process return code from status.
# WIFSIGNALED - True - Return True if the process returning status was terminated by a signal.
# WTERMSIG - 2 - Return the signal that terminated the process that provided the status value.
# WIFSTOPPED - False - Return True if the process returning status was stopped.
# WSTOPSIG - 0 - Return the signal that stopped the process that provided the status value.
```
See also: [Helpful things for
knight](https://github.com/xonsh/xonsh/pull/5361#issuecomment-2078826181).
### Before
```xsh
$RAISE_SUBPROC_ERROR = True
sleep 100
# Ctrl+C
_.rtn
# 0 # It's wrong and RAISE_SUBPROC_ERROR ignored.
for i in range(5):
print(i)
sleep 5
# 0
# Ctrl+C # Can't interrupt
# 1
# 2
```
### After
```xsh
sleep 100
# Ctrl+C
_.rtn
# -2 # Like in default Popen
$RAISE_SUBPROC_ERROR = True
for i in range(5):
print(i)
sleep 5
# 0
# Ctrl+C
# subprocess.CalledProcessError
```
### Notes
* We need to refactor `xonsh.jobs`. It's pretty uncomfortable work with
module.
* The logic is blurry between Specs, Pipelines and Jobs. We need to
bring more clear functions.
* The `captured` variable looks like "just the way of capturing (stdout,
object)" but in fact it affects all logic very much. We need to create
table where we can see the logic difference for every type of capturing.
E.g. in `captured=stdout` mode we use `xonsh.jobs` to monitor the
command but in `captured=object` we avoid this and some logic from
`xonsh.jobs` applied in `stdout` mode but missed in `object` mode. We
need clear map.
## For community
⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍
comment**
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* XONSH_TRACE_SUBPROC returns more useful details.
* update
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Name obj to thread because it's thread!
* update
* wip
* wip
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* wip
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Added XONSH_SUBPROC_OUTPUT_FORMAT
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
### Motivation
* To have an ability to manage the output format added ``$XONSH_SUBPROC_OUTPUT_FORMAT`` to switch the way to return the output lines. Default ``stream_lines`` to return text. Alternative ``list_lines`` to return the list of lines. Also supported custom lambda function.
* Additionally the [proposal to change default behavior](https://github.com/xonsh/xonsh/pull/5377#discussion_r1587627131) for a single line case.
* Closes#3924 as soft solution.
### Before
```xsh
mkdir -p /tmp/tst && cd /tmp/tst && touch 1 2 3
$(ls)
# '1\n2\n3\n'
id $(whoami)
# id: ‘pc\n’: no such user: Invalid argument
du $(ls)
# du: cannot access '1'$'\n''2'$'\n''3'$'\n': No such file or directory
ls $(fzf)
# ls: cannot access 'FUNDING.yml'$'\n': No such file or directory
```
### After
```xsh
mkdir -p /tmp/tst && cd /tmp/tst && touch 1 2 3
$XONSH_SUBPROC_OUTPUT_FORMAT = 'list_lines'
$(ls)
# ['1', '2', '3']
[f for f in $(ls)]
# ['1', '2', '3']
id $(whoami)
# uid=501(user) gid=20(staff)
du $(ls)
# 0 1
# 0 2
# 0 3
ls $(fzf)
# FUNDING.yml
# etc
mkdir -p /tmp/@($(whoami))/dir
cat /etc/passwd | grep $(whoami)
```
### Notes
* It will be good to improve parser for cases like `mkdir -p /tmp/$(whoami)/dir`. PR is welcome!
* I named the default mode as `stream_lines` (instead of just `stream` or `raw`) because in fact we transform raw output into stream of lines and possibly reduce the length of output ([replacing `\r\n` to `\n`](c3a12b2a9c/xonsh/procs/pipelines.py (L380-L383))). May be some day we need to add raw "stream" output format.
* Now anybody can implement bash `IFS` behavior in [bashisms](https://github.com/xonsh/xontrib-bashisms).
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
### Motivation
Basically we use `xonsh --no-rc` to have pure xonsh.
I noticed many times when parent environment can affect `xonsh --no-rc` e.g. `$XONSH_HISTORY_FILE`.
Current workaround is `env -i @$(which xonsh) --no-rc` but I think we need to have pure xonsh way.
Closes#4921
### Before
```xsh
xonsh --no-rc
$QWE = 1
xonsh --no-rc
$QWE
# 1
```
### After
```xsh
xonsh --no-rc
$QWE = 1
xonsh --no-rc --no-env
$QWE
# Unknown environment variable: $QWE
$XONSH_ENV_INHERITED # Marker to have a way to catch this situation.
# False
```
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Gil Forsyth <gforsyth@users.noreply.github.com>
* update
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: a <1@1.1>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
