'Bash script: read 30000 records from file and run multiple process in parallel

txt with more than 30000 records. All records are one for line and is an IP like this:

192.168.0.1
192.168.0.2
192.168.0.3
192.168.0.4
192.168.0.5
192.168.0.6
192.168.0.7
192.168.0.8
192.168.0.9
192.168.0.10

I read each row in a bash script, and I need to run a curl like this:

while IFS= read -r line || [[ -n "$line" ]]; do
    #check_site "$line"
    resp=$(curl -i -m1 http://$line 2>&1)
    echo "$resp" | grep -Eo "$ok" > /dev/null

    if [ $? -ne 0 ]; then
        #echo -e "failed: $line" >> "${logfile}"
        echo -e "Command: curl -i -m1 http://$line 2>&1" >> "${outfile}"
        echo -e "failed: $line:\n\n \"$resp\"\n\n" >> "${outfile}"
        echo "$line" >> "${faillog}"
    fi
done < "${FILE}"

Is there a method to run multiple lines simultaneously in my file to reduce the execution time?

linuxbashdo-whilescripttxt


Solution 1:[1]

I solved for the multiprocess in this way:

#export variable to be used into function    
export outlog="/tmp/out.log"
export faillog="/tmp/fail.log"
export ok="(curl: \(7\) Failed to connect to)" # acceptable responses

# create function:
check_site() {
  ip=$1
  resp=$(curl -i -m1 http://$ip 2>&1)
  echo "$resp" | grep -Eo "$ok" > /dev/null

  if [ $? -ne 0 ]; then
    echo -e "Command: curl -i -m1 http://$ip 2>&1" >> "${outlog}"
    echo -e "Block failed: $ip:\n\n \"$resp\"\n\n" >> "${outlog}"
    echo "$ip" >> "${faillog}"
  fi
}

# call the function:
export -f check_site
parallel -j 252 -a "${FILE}" check_site

Solution 2:[2]

Xargs will do the trick. Wikipedia

Solution 3:[3]

This article describe approach to resolve parallel execution, it may help you: Parallel execution in Bash

Example from the article:

#!/bin/bash

RANDOM=10
JOBS_COUNTER=0
MAX_CHILDREN=10
MY_PID=$$

for i in {1..100}
do
    echo Cycle counter: $i
    JOBS_COUNTER=$((`ps ax -Ao ppid | grep $MY_PID | wc -l`))
    while [ $JOBS_COUNTER -ge $MAX_CHILDREN ]
    do
        JOBS_COUNTER=$((`ps ax -Ao ppid | grep $MY_PID | wc -l`))
        echo Jobs counter: $JOBS_COUNTER
        sleep 1
    done
    sleep $(($RANDOM % 30)) &
done
echo Finishing children ...
# wait for children here
while [ $JOBS_COUNTER -gt 1 ]
do
    JOBS_COUNTER=$((`ps ax -Ao ppid | grep $MY_PID | wc -l`))
    echo Jobs counter: $JOBS_COUNTER
    sleep 1
done
echo Done

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1
Solution 2
Solution 3 Podrepny

Related Questions