'How to fix “No newline at end of file” warning for lots of files?
I have a huge number of source files that are all lacking a newline at the end.
How do I automatically add a newline to the end of each of them?
Some may already have a newline, so it should only be added if necessary.
I'm probably not looking for code, per se, but just something I can run in Terminal to add the necessary newlines (or some kind of programming or development tool).
Solution 1:[1]
Converted Norman's answer to a split one-liner for convenience.
for i in * ; do echo $i; \
if diff /dev/null "$i" | tail -1 | \
grep '^\\ No newline' > /dev/null; then echo >> "$i"; \
fi; done
Replace * with whatever file pattern you want, eg *.c
And another to just tell you which files are broken:
for i in * ; do \
if diff /dev/null "$i" | tail -1 | \
grep '^\\ No newline' > /dev/null; then echo $i; \
fi; done
Solution 2:[2]
If you have access to Unix tools, you can run diff
to find out which files lack a newline and then append it:
#!/bin/sh
for i
do
if diff /dev/null "$i" | tail -1 | grep '^\\ No newline' > /dev/null
then
echo >> "$i"
fi
done
I'm relying on diff
to produce the message with a \
in the first column, tail
to give me the last line of diff
's output, and grep
to tell me if the last line is the message I'm looking for. If all that works, then the echo
produces a newline and the >>
appends it to the file "$i"
. The quotes around "$i"
make sure things still work if the filename has spaces in it.
Solution 3:[3]
A simple fix for files that are "missing" newline at end of file is simply sed; the following fixes the file "in-place" (using the "-i" option):
find . -type f -exec sed -i -e '$a\' {} \; -print
Explanation:
- find all files (
-type f
), - run
sed
, - change the files in-place (
-i
), - given the following (
-e
) script/expression, which matches the end of the file ($
),- and perform the "append" action (
a\
), - but don't actually specify any text to append (nothing after the
\
) which is going to add a newline to the end of the file, but only if it's missing.
- and perform the "append" action (
- Prints all files found (fixed or not), which is probably unnecessary.
Main caveat is that sed
features vary across platforms, so -i
and -e
may or may not be supported / the same; e.g. older Unix, or MacOS oddities may require slightly different syntax.
To only operate on filename(s) matching specific suffix(es), just add find path/to/dir -type f \( -name \*.C -o -name \*.h -o -name \*.java \) -exec ...
Solution 4:[4]
OK, after complaining in the comments, there is my better solution. First, you want to know, which files are missing newlines:
find -type f -exec sh -c "tail -1 {} | xxd -p | tail -1 | grep -v 0a$" ';' -print
Not super fast (calling a couple of processes for each file), but it's OK for practical use.
Now, when you have it, you may as well add the newline, with another -exec
:
find -type f -exec sh -c "tail -1 {} | xxd -p | tail -1 | grep -v 0a$" ';' -exec sh -c "echo >> {}" ';'
Possible gotchas:
if filenames are bad, e.g. they have spaces, you may need
tail -1 \"{}\"
. Or does find do it right?you may want to add more filtering to find, like
-name \*py
, or the like.think about possible DOS/Unix newlines mess before use (fix that first).
EDIT:
If you don't like the output from these commands (echoing some hex), add -q
to grep:
find -type f -exec sh -c "tail -1 {} | xxd -p | tail -1 | grep -q -v 0a$" ';' -print
find -type f -exec sh -c "tail -1 {} | xxd -p | tail -1 | grep -q -v 0a$" ';' -exec sh -c "echo >> {}" ';'
Solution 5:[5]
I'm surprised nobody has mentioned that many simple text-processing tools like Awk will add a newline as a side effect. Here is a simple loop which will overwrite a file only if a newline was actually added.
for f in *; do
awk 1 "$f" >tmp
cmp -s tmp "$f" || mv tmp "$f"
done
rm -f tmp
(The temporary file is obviously a bit of a wart.)
IDEone demo: http://ideone.com/HpRHcx
Solution 6:[6]
Try ex-way:
ex -s +"bufdo wq" *.c
And recursively (with a new globbing option enabled):
ex -s +"bufdo wq" **/*.c
This is equivalent to vi -es
. Change *.c
to extension of your interest.
The ex
/vi
would automatically append newline on save if it's not present.
Solution 7:[7]
find -type f | while read f; do [[ `tail -c1 "$f"` ]] && echo >> "$f"; done
I'm using find
instead of for f in *
as it is recursive and the question was about "huge number of source files".
I'm using while read
instead of find -exec
or xargs
for performance reasons, it saves spawning shell process every time.
I'm taking advantage of the fact that backtick operator is returning output of command "with any trailing newlines deleted" man bash
, so for properly terminated files backtick will be empty and echo will be skipped.
The find | read
couple will fail on filenames that contain newlines, but it's easy to fix if required:
find -type f -print0 | while read -d $'\0' f; do [[ `tail -c1 "$f"` ]] && echo >> "$f"; done
Solution 8:[8]
Below is my bash script solution. It first checks that the file is a text file. Then, if it's a text file, it uses tail and od (octal dump) to see if the last character is the newline character. If it isn't, then it appends a newline using echo:
item="$1"
if file "$item" | egrep '\btext\b' > /dev/null
then
if ! tail -c 1 "$item" | od -b -A n | egrep '\b012\b' > /dev/null
then
echo "(appending final newline to ${item})"
echo >> "$item"
fi
fi
Solution 9:[9]
Due To command localization Tim and Norman answer Shall be improved using 'LANG=C' prefix to have a chance to match 'No newline' pattern with every system having any regional parameters
This ensures an ending empty line to every file put on the command line of this script :
#!/bin/sh -f
for i in $* ; do echo $i; \
if LANG=C diff /dev/null "$i" | tail -1 | \
grep '^\\ No newline' > /dev/null; then echo >> "$i"; \
fi; done
And this script detects files lacking of it :
#!/bin/sh -f
for i in $* ; do \
if LANG=C diff /dev/null "$i" | tail -1 | \
grep '^\\ No newline' > /dev/null; then echo $i; \
fi; done
Solution 10:[10]
After finding the tool do this job with no luck. I decide to write my own
This is my python script to do that job
It only append (\r\n) to file not contains (\n) at the end of file
https://github.com/tranhuanltv/append_newline
Usage: append_newline.py .c ./projects ./result_dir
Make Pull Requests if you want to
Solution 11:[11]
pcregrep --recursive --exclude-dir=.git \
--files-without-match --multiline '\n\z' . |
while read k ; do echo >> "$k"; done
There are several steps involved here:
- Recursively find files
- Detect which files lack a trailing new line
- Loop over each of those files
- Append the newline
Step 1 is traditionally done with find
(following the Unix tradition of
"each tool doing one thing and doing it well"), but since pcregrep has builtin support, I'm comfortable using it. I'm careful to avoid messing around with the .git folder.
Step 2 is done with a multiline regular expression matching files that do have a final newline, and printing the names of files that don't match.
Step 3 is done with a while/read loop rather than a for/in, since the latter fails for filenames with spaces and for extremely long lists of files.
Step 4 is a simple echo, following @norman-ramsey's approach.
h/t @anthony-bush https://stackoverflow.com/a/20687956/577438 for the pcregrep suggestion.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|---|
Solution 1 | |
Solution 2 | |
Solution 3 | |
Solution 4 | |
Solution 5 | tripleee |
Solution 6 | |
Solution 7 | Kamil Christ |
Solution 8 | |
Solution 9 | |
Solution 10 | Tran Huan |
Solution 11 | Community |