How to detect and remove indentation of a piped text

Question

I'm looking for a way to remove the indentation of a piped text. Below is a solution using cut -c 9- which assumes the indentation is 8 character wide.

I'm looking for a solution which can detect the number of spaces to remove. This implies going through the whole (piped) file to know the minimum number of spaces (tabs?) used to indent it, then remove them on each line.

run.sh

help() {
    awk '
    /esac/{b=0}
    b
    /case "\$arg" in/{b=1}' \
    "$me" \
    | cut -c 9-
}

while [[ $# -ge 1 ]]
do
    arg="$1"
    shift
    case "$arg" in
        help|h|?|--help|-h|'-?')
            # Show this help
            help;;
    esac
done

$ ./run.sh --help

help|h|?|--help|-h|'-?')
    # Show this help
    help;;

---

Note: echo $' 4\n 2\n 3' | python3 -c 'import sys; import textwrap as tw; print(tw.dedent(sys.stdin.read()), end="")' works but I expect there is a better, way (I mean, one which doesn't only depends on software more common than python. Maybe awk? I wouldn't mind seeing a perl solution either.

Note2: echo $' 4\n 2\n 3' | python -c 'import sys; import textwrap as tw; print tw.dedent(sys.stdin.read()),' also works (Python 2.7.15rc1).

Answer 1

The following is pure bash, with no external tools or command substitutions:

#!/usr/bin/env bash
all_lines=( )
min_spaces=9999 # start with something arbitrarily high
while IFS= read -r line; do
  all_lines+=( "$line" )
  if [[ ${line:0:$min_spaces} =~ ^[[:space:]]*$ ]]; then
    continue  # this line has at least as much whitespace as those preceding it
  fi
  # this line has *less* whitespace than those preceding it; we need to know how much.
  [[ $line =~ ^([[:space:]]*) ]]
  line_whitespace=${BASH_REMATCH[1]}
  min_spaces=${#line_whitespace}
done

for line in "${all_lines[@]}"; do
  printf '%s\n' "${line:$min_spaces}"
done

Its output is:

  4
2
 3

Answer 2

Suppose you have:

$ echo $'    4\n  2\n   3\n\ttab'
    4
  2
   3
    tab

You can use the Unix expand utility to expand the tabs to spaces. Then run through an awk to count the minimum number of spaces on a line:

$ echo $'    4\n  2\n   3\n\ttab' | 
expand | 
awk 'BEGIN{min_indent=9999999}
     {lines[++cnt]=$0
      match($0, /^[ ]*/)
      if(RLENGTH<min_indent) min_indent=RLENGTH
     }
     END{for (i=1;i<=cnt;i++) 
               print substr(lines[i], min_indent+1)}'
  4
2
 3
      tab

Answer 3

Here's the (semi-) obvious temp file solution.

#!/bin/sh

t=$(mktemp -t dedent.XXXXXXXXXX) || exit
trap 'rm -f $t' EXIT ERR
awk '{ n = match($0, /[^ ]/); if (NR == 1 || n<min) min = n }1
    END { exit min+1 }' >"$t"
cut -c $?- "$t"

This obviously fails if all lines have more than 255 leading whitespace characters because then the result won't fit into the exit code from Awk.

This has the advantage that we are not restricting ourselves to the available memory. Instead, we are restricting ourselves to the available disk space. The drawback is that disk might be slower, but the advantage of not reading big files into memory will IMHO trump that.

Answer 4

echo $'    4\n  2\n   3\n  \n   more spaces in  the    line\n  ...' | \
(text="$(cat)"; echo "$text" \
| cut -c "$(echo "$text" | sed 's/[^ ].*$//' | awk 'NR == 1 {a = length} length < a {a = length} END {print a + 1}')-"\
)

With explanations:

echo $'    4\n  2\n   3\n  \n   more spaces in  the    line\n  ...' | \
(
    text="$(cat)" # Obtain the input in a varibale
    echo "$text" | cut -c "$(
        # `cut` removes the n-1 first characters of each line of the input, where n is:
            echo "$text" | \
            sed 's/[^ ].*$//' | \
            awk 'NR == 1 || length < a {a = length} END {print a + 1}'
            # sed: keep only the initial spaces, remove the rest
            # awk:
            # At the first line `NR == 1`, get the length of the line `a = length`.
            # For any shorter line `a < length`, update the length `a = length`.
            # At the end of the piped input, print the shortest length + 1.
            # ... we add 1 because in `cut`, characters of the line are indexed at 1.
        )-"
)

---

Update:

It is possible to avoid spawning sed. As per tripleee's comment, sed's s/// can be replace awk's sub(). Here is an even shorter option, using n = match() as in tripleee's answer.

echo $'    4\n  2\n   3\n  \n   more spaces in  the    line\n  ...' | \
(
    text="$(cat)" # Obtain the input in a varibale
    echo "$text" | cut -c "$(
        # `cut` removes the a-1 first characters of each line of the input, where a is:
            echo "$text" | \
            awk '
                {n = match($0, /[^ ]/)}
                NR == 1 || n < a {a = n}
                END || a == 0 {print a + 1; exit 0}'
            # awk:
            # At every line, get the position of the first non-space character
            # At the first line `NR == 1`, copy that lenght to `a`.
            # For any line with less spaces than `a` (`n < a`) update `a`, (`a = n`).
            # At the end of the piped input, print a + 1.
            # a is then the minimum number of common leading spaces found in all lines.
            # ... we add 1 because in `cut`, characters of the line are indexed at 1.
            #
            # I'm not sure the whether the `a == 0 {...;  exit 0}` optimisation will let the "$text" be written to the script stdout yet (which is not desirable at all). Gotta test that when I get the time.

        )-"
)

Apparently, it's also possible to do in Perl 6 with the function my &f = *.indent(*);.

Answer 5

Another solution with awk, based on dawg’s answer. Major differences include:

• No need to set an arbitrary large number for indentation, which feels hacky.
• Works on text with empty lines, by not considering them when gathering the lowest indented line.

awk '
  {
    lines[++count] = $0
    if (NF == 0) next
    match($0, /[^ ]/)
    if (length(min) == 0 || RSTART < min) min = RSTART
  }
  END {
    for (i = 1; i <= count; i++) print substr(lines[i], min)
  }
' <<< $'    4\n  2\n   3'

Or all on the same line

awk '{ lines[++count] = $0; if (NF == 0) next; match($0, /[^ ]/); if (length(min) == 0 || RSTART < min) min = RSTART; } END { for (i = 1; i <= count; i++) print substr(lines[i], min) }' <<< $'    4\n  2\n   3'

Explanation:

Add current line to an array, and increment count variable

{
  lines[++count] = $0

If line is empty, skip to next iteration

  if (NF == 0) next

Set RSTART to the start index of the first non-space character.

  match($0, /[^ ]/)

If min isn’t set or is higher than RSTART, set the former to the latter.

  if (length(min) == 0 || RSTART < min) min = RSTART
}

Run after all input is read.

END {

Loop over the array, and for each line print only a substring going from the index set in min to the end of the line.

  for (i = 1; i <= count; i++) print substr(lines[i], min)
}

Answer 6

solution using bash

#!/usr/bin/env bash
cb=$(xclip -selection clipboard -o)
firstchar=${cb::1}
if [ "$firstchar" == $'\t' ];then
  tocut=$(echo "$cb" | awk -F$'\t' '{print NF-1;}' | sort -n | head -n1)
else
  tocut=$(echo "$cb" | awk -F '[^ ].*' '{print length($1)}' | sort -n | head -n1)
fi

echo "$cb" | cut -c$((tocut+1))- | xclip -selection clipboard

Note: assumes first line has the left-most indent

Works for both spaces and tabs

Ctrl+V some text, run that bash script, and now the dedented text is saved to your clipboard

solution using python

detab.py

import sys
import textwrap

data = sys.stdin.readlines()
data = "".join(data)
print(textwrap.dedent(data))

use with pipes

xclip -selection clipboard -o | python detab.py | xclip -selection clipboard