'How to prevent yq removing comments and empty lines?
Here Edit yaml objects in array with yq. Speed up Terminalizer's terminal cast (record) I asked about how to edit yaml with yq. I received the best answer. But by default yq
removes comments and empty lines. How to prevent this behavior?
input.yml
# Specify a command to be executed
# like `/bin/bash -l`, `ls`, or any other commands
# the default is bash for Linux
# or powershell.exe for Windows
command: fish -l
# Specify the current working directory path
# the default is the current working directory path
cwd: null
# Export additional ENV variables
env:
recording: true
# Explicitly set the number of columns
# or use `auto` to take the current
# number of columns of your shell
cols: 110
execute
yq -y . input.yml
result
command: fish -l
cwd: null
env:
recording: true
cols: 110
Solution 1:[1]
In some limited cases you could use diff/patch along with yq.
For example if input.yml
contains your input text, the commands
$ yq -y . input.yml > input.yml.1
$ yq -y .env.recording=false input.yml > input.yml.2
$ diff input.yml.1 input.yml.2 > input.yml.diff
$ patch -o input.yml.new input.yml < input.yml.diff
creates a file input.yml.new
with comments preserved but
recording changed to false:
# Specify a command to be executed
# like `/bin/bash -l`, `ls`, or any other commands
# the default is bash for Linux
# or powershell.exe for Windows
command: fish -l
# Specify the current working directory path
# the default is the current working directory path
cwd: null
# Export additional ENV variables
env:
recording: false
# Explicitly set the number of columns
# or use `auto` to take the current
# number of columns of your shell
cols: 110
Solution 2:[2]
This is improvement of How to prevent yq removing comments and empty lines? comment.
In mine case was no enough diff -B
and diff -wB
as it still does not keep blank lines and keep generate an entire file difference as a single chunk instead of many small chunks.
Here is example of the input (test.yml
):
# This file is automatically generated
#
content-index:
timestamp: 1970-01-01T00:00:00Z
entries:
- dirs:
- dir: dir-1/dir-2
files:
- file: file-1.dat
md5-hash:
timestamp: 1970-01-01T00:00:00Z
- file: file-2.dat
md5-hash:
timestamp:
- file: file-3.dat
md5-hash:
timestamp:
- dir: dir-1/dir-2/dir-3
files:
- file: file-1.dat
md5-hash:
timestamp:
- file: file-2.dat
md5-hash:
timestamp:
If try to edit a field and generate the difference file:
diff -B test.yml <(yq -y ".\"content-index\".timestamp=\"2022-01-01T00:00:00Z\"" test.yml)
It does keep remove blank lines:
5,7c2
<
< timestamp: 1970-01-01T00:00:00Z
<
---
> timestamp: '2022-01-01T00:00:00Z'
Adds everywhere null
instead of an empty field and changes the rest of timestamp fields (which means you have to use '...' to retain these as is):
17,19c8,9
< md5-hash:
< timestamp: 1970-01-01T00:00:00Z
<
---
> md5-hash: null
> timestamp: '1970-01-01T00:00:00+00:00'
The -wB
flags changes the difference file from a single chunk into multiple chunks, but still does remove blank lines.
Here is a mention of that diff issue: https://unix.stackexchange.com/questions/423186/diff-how-to-ignore-empty-lines/423188#423188
To fix that you have to use it with grep:
diff -wB <(grep -vE '^\s*$' test.yml) <(yq -y ".\"content-index\".timestamp=\"2022-01-01T00:00:00Z\"" test.yml)
But nevertheless it still does remove comments:
1,2d0
< # This file is automatically generated
< #
Here is solution for that: https://unix.stackexchange.com/questions/17040/how-to-diff-files-ignoring-comments-lines-starting-with/17044#17044
So the complete oneliner is:
diff -wB <(grep -vE '^\s*(#|$)' test.yml) <(yq -y ".\"content-index\".timestamp=\"2022-01-01T00:00:00Z\"" test.yml) | patch -o - test.yml 2>/dev/null
Where 2>/dev/null
stands to ignore patch warnings like:
Hunk #1 succeeded at 6 (offset 4 lines).
To avoid it in real code, you can use the -s
flag instead:
... | patch -s -o ...
Update:
There is a better complete implementation as a shell script for GitHub Actions pipeline composite action.
GitHub Composite action: https://github.com/andry81-devops/gh-action--accum-content
Bash scripts:
https://github.com/andry81-devops/gh-workflow/blob/master/bash/cache/accum-content.sh
https://github.com/andry81-devops/gh-workflow/blob/master/bash/github/init-yq-workflow.sh
The implementation can use 2 of yq
implementations:
- https://github.com/kislyuk/yq - a
jq
wrapper (default Cygwin distribution) - https://github.com/mikefarah/yq - Go implementation (default Ubuntu 20.04 distribution)
Search for: yq_edit
, yq_diff
, yq_patch
functions
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|---|
Solution 1 | jq170727 |
Solution 2 |