Category "awk"

AWK remove query params from URL

I have access.log file with >1m lines. The exaple of line: 113.10.154.38 - - [27/May/2016:03:36:26 +0200] "POST /index.php?option=com_jce&task=plugin&

select subdomains using print command

cat a.txt a.b.c.d.e.google.com x.y.z.google.com rev a.txt | awk -F. '{print $2,$3}' | rev This is showing: e google x google But I want this output a.b.c.d.e

How to substitute alternate positions in a string using awk

I have a lookup file that I use to search the available records in file_2 and if such records are present then replace those records with #. Currently my code i

awk split on first occurrence of character

Trying to use awk to split each line. If there is more the one p or q the second split on the ( does not work correctly (line 2 is an example. I am not able to

Standard Deviation from multiple files in bash

I wish to calculate the standard deviation from a range of files titled "res_NUMBER.cs" which are formatted as a CSV. Example data includes 1,M,CA,54.9130 1,M

Change indented multiline string in file in terminal

i want to change multiline string within particular marks that is indented in file, example looks like this: non_important_array: [ text ], impo

What's the most robust way to efficiently parse CSV using awk?

The intent of this question is to provide a canonical answer. Given a CSV as might be generated by Excel or other tools with embedded newlines and/or double quo

how to clear map in awk?

I have a file a.txt. There are many numbers in each line in a.txt: 1 2 3 1 2 5 6 7 7 19 20 20 I have to print each line without duplicate numbers, just like

rename files that match strings in a txt file

I am trying to rename multiple files according to the match to a .txt file my files are GCF_000698265.1_ASM69826v1_genomic.gff.gz GCF_000785125.1_ASM78512v1_gen

How to sum the values in one data column of duplicate rows with awk and remove the duplicate rows?

I have a CSV file with a heading row and multiple data rows each with 11 data columns like this: Order Date,Username,Order Number,No Resi,Quantity,Title,Update

Error from server (NotFound): pods "$(kubectl" not found

In the minikube hyper-v machine I did deployment of sawtooth-0, using sawtooth config file. Now when I am checking running pods it seems to have some default o

Ignore header when performing awk command - not working

I know several questions have been asked about ignoring the header using the awk command, but I have tried a bunch of the suggested methods and it still isn't w

How to split a YAML into multiple files with a proper name

I have a valid YAML: --- name: first metadata: a --- name: second metadata: b --- name: third metadata: c How can I split it using a one-liner AWK script in

How to rename key/header in csv DictReader

For instance given mycsv.csv file h1,h2 a,b c,d how to rename h2 onto HTwo with reader reader = csv.DictReader(open('mycsv.csv')) (Additionally how to wr

Using curl in a bash script and getting curl: (3) Illegal characters found in URL

So I have a very simple bash script that is curl'ing to an auth server for a header. The header url is written to a var and then used in the next curl call. Whe

Check for multi-line content in a file

I'm trying to check if a multi-line string exists in a file using common bash commands (grep, awk, ...). I want to have a file with a few lines, plain lines, n

Processing text in bash - extracting the volume of a program from pactl sink-inputs output

Looking for a way to extract the volume from pactl list sink-inputs Output example: Sink Input #67 Driver: protocol-native.c Owner Module: 1

Default field separator for awk

Is the default separator only space for awk?

How to select two columns from awk and print if they do not match

I need to select two MSISDN values from OMO account Migration logs and print the ones that do not match. [2019-03-11 04:15:08 INFO-SUBAPP ESBRestClient:117] ##

Finding second highest salary using awk

I have a file as follows 1 rob hr 10000 2 charls it 20000 4 kk Fin 30000 5 km it 30000 6 kl it 30000 7 mark hr 10000 8 kc it 30000 9 dc fin 40000 10 m