How to process pathnames with ! within an for /F loop?

Question

In a complex batch file I want to read in files with paths, among other things, to read them into a variable one after the other separated by spaces.

This works with the following code so far quite well - but only if the path does not contain an exclamation mark.

Even using the setlocal command (enabledelayedexpansion / disabledelayedexpansion) I did not succeed in processing exclamation marks.

Does anyone here have a clever idea to the problem?

The following example batch creates a text file in the current directory and then reads it in a for /F loop. At the end all three paths from the text file should be in the variable %Output%. But with the exclamation mark.

@echo off

setlocal enabledelayedexpansion

echo This is an example^^! > "textfile.txt"
echo This is a second example^^! >> "textfile.txt"
echo And this line have an ^^! exclamation mark in the middle >> "textfile.txt"

for /F "usebackq tokens=* delims=" %%a in (textfile.txt) do (
    set "Record=%%a"
    set "Output=!Output!!Record! - "
  )
)

echo %Output%
echo !Output!

endlocal

The Output is like this:

This is an example  - This is a second example  - And this line have an  exclamation mark in the middle

But should be like this:

This is an example!  - This is a second example!  - And this line have an ! exclamation mark in the middle

Answer 1

It is advisable not using delayed variable expansion on processing files and directories, lines in a text file, strings not defined by the batch file itself, or output captured from the execution of a program or a command line. If it is for some reasons necessary to make use of delayed variable expansion inside a FOR loop, there should be first assigned the file/directory name, the line, or the string to process to an environment variable while delayed expansion is disabled and then enable delayed expansion temporary inside the FOR loop.

Here is a batch file demo which can be simply run from within a command prompt window or by double clicking on the batch file. It creates several files for demonstration in the directory for temporary files, but deletes them all before exiting.

@echo off
setlocal EnableExtensions DisableDelayedExpansion

echo This is an example!> "%TEMP%\TextFile.tmp"
echo This is a second example!>> "%TEMP%\TextFile.tmp"
echo And this line has an exclamation mark ! in the middle.>> "%TEMP%\TextFile.tmp"

set "Output="
(for /F usebackq^ delims^=^ eol^= %%I in ("%TEMP%\TextFile.tmp") do set "Line=%%I" & call :ConcatenateLines) & goto ContinueDemo
:ConcatenateLines
set "Output=%Output% - %Line%" & goto :EOF

:ContinueDemo
cls
echo/
echo All lines concatenated are:
echo/
echo %Output:~3%
set "Output="
del "%TEMP%\TextFile.tmp"

echo File with name ".Linux hidden file!">"%TEMP%\.Linux hidden file!"
echo File with name "A simple test!">"%TEMP%\A simple test!"
echo File with name "  100%% Development & 'Test' (!).tmp">"%TEMP%\  100%% Development & 'Test(!)'.tmp"
echo/
echo Files with ! are:
echo/
for /F "eol=| tokens=* delims=" %%I in ('dir "%TEMP%\*!*" /A-D /B /ON 2^>nul') do (
    set "NameFile=%%I"
    set "FileName=%%~nI"
    set "FileExtension=%%~xI"
    set "FullName=%TEMP%\%%I"
    setlocal EnableDelayedExpansion
    if defined FileName (
        if defined FileExtension (
            echo File with ext. !FileExtension:~1!: !NameFile!
        ) else (
            echo Extensionless file: !NameFile!
        )
    ) else echo Extensionless file: !NameFile!
    del "!FullName!"
    endlocal
)
endlocal
echo/
@setlocal EnableExtensions EnableDelayedExpansion & for /F "tokens=1,2" %%G in ("!CMDCMDLINE!") do @endlocal & if /I "%%~nG" == "cmd" if /I "%%~H" == "/c" set /P "=Press any key to exit the demo . . . "<nul & pause >nul

The output of this batch file is:

All lines concatenated are:

This is an example! - This is a second example! - And this line has an exclamation mark ! in the middle.

Files with ! are:

File with ext. tmp:   100% Development & 'Test(!)'.tmp
Extensionless file: .Linux hidden file!
Extensionless file: A simple test!

The text file example with concatenating lines makes use of a subroutine called from within the FOR loop processing the lines in the text file. The syntax used here is for maximum performance by getting the subroutine as near as possible to the FOR command line. That is important if the FOR loop has to process hundreds or even thousands of items.

The example processing file names enables and disables delayed expansion inside the FOR loop after having assigned all parts of the currently processed file to environment variables. It could be useful to reduce the list of environment variables before processing thousands of files for a better performance on using this method.

Another method is shown in Magoo´s answer using the command CALL to get a command line with referenced environment variables (re)defined inside the loop parsed a second time. I used that method also in the past quite often, but don't that anymore as it is not fail-safe and not efficient. call set results in searching by cmd.exe in current directory and next in all directories of environment variable PATH for a file with name set and a file extension of environment variable PATHEXT. So it results in lots of file system accesses in the background on each iteration of the FOR loop and if there is by chance a file set.exe, set.bat, set.cmd, etc. found by cmd.exe somewhere, the batch file does not work anymore as expected because of running the executable or calling the batch file instead of the (re)definition of the environment variable.

The following answers written by me could be also helpful:

• How to read and print contents of text file line by line?
  It explains in full details how to process all lines of a text file.
• How to pass environment variables as parameters by reference to another batch file?
  It explains in full details what the commands SETLOCAL and ENDLOCAL do.
• How to pass a command that may contain special characters (such as % or !) inside a variable to a for /f loop?
  This is an example of a batch file designed to process video files with any valid file name on any Windows computer very efficient, safe and secure with full explanation.

Answer 2

Well, the main trick is to enable delayed expansion only when it is actually needed and to disable it otherwise. Since you are accumulating multiple strings in a single variable inside of a loop, it becomes a bit more difficult, because you should have delayed expansion disabled during expansion of for meta-variables (like %%a), but enabled when joining the string, leading to setlocal and endlocal statements inside of the loop. The major purpose of these commands is environment localisation, hence any variable changes become lost past endlocal, so a method of tansfering the value beyond endlocal is required, which is incorporated in the following code:

@echo off
setlocal EnableExtensions DisableDelayedExpansion

rem /* At this point delayed expansion is disabled, so there is no need to
rem    escape exclamation marks; moreover a redirected block prevents
rem    superfluous file close/reopen operations, and there is no more
rem    trailing space written to the file (as in your original approach): */
> "textfile.txt" (
    echo This is an example!
    echo This is a second example!
    echo And this line have an ! exclamation mark in the middle
)

rem // Let us initialise the output variable:
set "Output= - "
rem // Using `usebackq` only makes sense when you want to quote a file path:
for /F "usebackq tokens=* delims=" %%a in ("textfile.txt") do (
    rem // Remember that delayed expansion is still disabled at this point:
    set "Record=%%a"
    rem // For concatenation we need delayed expansion to be enabled:
    setlocal EnableDelayedExpansion
    set "Output=!Output!!Record! - "
    rem /* We need to terminate the environment localisation of `setlocal`
    rem    inside of the loop, but we would lose any changes in `Output`;
    rem    therefore let us (mis-)use `for /F`, which is iterated once: */
    for /F "delims=" %%b in ("!Output!") do endlocal & set "Output=%%b"
    rem /* An often used method to transfer a variable beyond `endlocal` is
    rem    the line `endlocal & set "Output=%Output%`, but this only works
    rem    outside of a parenthesised block because of percent expansion. */
)

rem /* Echo out text with delayed expansion enabled is the only safe way;
rem    surrounding separators ` - ` are going to be removed; since `Output`
rem    was initialised with something non-empty, we do not even need to skip
rem    sub-string expansion for the problematic case of an empty string: */
setlocal EnableDelayedExpansion
echo(!Output:~3,-3!
endlocal

endlocal
exit /B

Answer 3

Pew. I finally got it to work. It works via a workaround using a second text file. Not pretty, not performant, but it works and is sufficient for my purposes. @Magoo, thanks for your post.

This is my solution:

@echo off
setlocal enableextensions enabledelayedexpansion

echo This is an example^^!> "textfile.txt"
echo This is a second example^^!>> "textfile.txt"
echo And this line have an ^^! exclamation mark in the middle>> "textfile.txt"
echo.
echo Content of the textfile:
type "textfile.txt"

set output=

del "textfile2.txt" 1> nul 2>&1
setlocal disabledelayedexpansion

for /f "usebackq tokens=* delims=" %%a IN ("textfile.txt") do (
  rem Write each line without a newline character into a new text file
  echo|set /p "dummy=%%a, ">>"textfile2.txt"
)

endlocal

rem Loading the content of the new text file into the variable
set /p output=<"textfile2.txt"
del "textfile2.txt" 1> nul 2>&1

echo.
echo --------------------------------------------
echo Content of the variable:

set out

endlocal

The output looks like this:

Content of the textfile:
This is an example!
This is a second example!
And this line have an ! exclamation mark in the middle

--------------------------------------------
Content of the variable:
output=This is an example!, This is a second example!, And this line have an ! exclamation mark in the middle,

Answer 4

It's delayedexpansion mode that appears to raise this problem.

@ECHO OFF
setlocal enabledelayedexpansion

echo This is an example^^^! > "textfile.txt"
echo This is a second example^^^! >> "textfile.txt"
echo And this line have an ^^^! exclamation mark in the middle >> "textfile.txt"
TYPE "textfile.txt"
SETLOCAL disabledelayedexpansion
for /F "usebackq tokens=* delims=" %%a in (textfile.txt) do (
    set "Record=%%a"
    CALL set "Output2=%%Output2%%%%record%% - "
    CALL set "Output=%%Output%%%%a - "
    SET out
  )
)
endlocal&SET "output=%output%"

echo %Output%
echo !Output!
SET out

I've no doubt that with delayedexpansion off, there would be the same problem with %. Just special characters, I suppose.

Note that with endlocal&SET "output=%output%", the set is executed in delayedexpansion mode.