Find the instances of each of a list of strings and print each set in a separate file

Giving -1 1 and -2 1 as options to join is useless: 1 is their default argument. I also doubt you want -a 1, which makes no difference on the given input. You want to give -k 2,2 to sort, not -k 2 (if not all addresses would start with two digits, the output would be wrong). '-' in argument of awk is useless. I am getting tired of always telling you the exact same things...

As far as I understand what you are trying to achieve, any output file would always have the same PTR repeated in the first column and the same "number of instances" (whatever that is; it is not the number of addresses) in the last column. That is a waste of disk space. Also the last sort in you command line becomes useless (the "number of instances" being always the same in a file). The command line below only writes addresses in files whose names are the "number of instances", followed by a comma, followed by the PTR:

$ mkdir out; sort -u IPv4.May2020.37.nMapoG.txt | awk 'FILENAME == ARGV[1] { a[$1] = $2 } FILENAME == ARGV[2] && $1 in a { print $2 >> "out/" a[$1] "," $1 }' PTRList.txt -

Because the "number of instances" starts the name of a file, 'ls -v out' sorts the file using that number (and so do most file managers).

All that said, using many small files is usually a terrible idea, performance-wise. You may want to use the output of your join (minus its useless options).

Would I pick -a 2 in my join file, the output file would be twenty times as big as the
file output with -a 1, and I would have a great deal of editing to do to clean it up.
Yes, either -a 1 or -a 2 ultimately gives the same net output. I prefer the succinct one.

I suspect that
$ mkdir out; sort -u IPv4.May2020.37.nMapoG.txt | awk 'FILENAME == ARGV[1] { a[$1] = $2 } FILENAME == ARGV[2] && $1 in a { print $2 >> "out/" a[$1] "," $1 }' PTRList.txt -

Is meant to produce something similar to my preamble script:
join -a 1 -1 1 -2 1 <(sort PTRList.txt) <(sort -u IPv4.May2020.37.nMapoG.txt) | sort -nrk 2 | awk '{print $1"\t"$3"\t"$2}' '-' > Temp.0716.01.txt

but I haven't managed to figure out how to state the two require arguments in Magic Banana's script.

The actual IPv4.May2020.37.nMapoG.txt will be a concatenation of 38 nmap output files,
each one 60MB to 80MB, totalling about 3.5GB, and containing about 1GB of No_DNS data
that would be better left out of the results.

George Langford

As I've stated before, there are redundancies in my scripts, but they force me to
check the order of the columns in the original files. Once the script functions as
I expect it should, I relax and press on. Time flies when one's making progress.

The whole exercise in this posting is exactly to create a whole mess of files,
each one listing the (sometimes millions) of IP addresses claiming the same PTR
record. These text files will be linked to the Recent Visitor data spreadsheets
(which Magic Banana has helpfully taught me to code in HTML), thereby keeping the
basic presentation within reason. The entire set of illustrative address listings
would make one webpage hopelessly immense, defeating the purpose of calling attention
to the abuse of IPv4 and IPv6 address space.

The address listings ought to remain in text form so that exploring the contents of
those many like-named servers is done only by experienced and wary observers.

The number-of-instances column in PTRList.txt is retained only as a check on the
accuracy of any script that creates individual text files for all the multi-address
PTR records. Concatenated into one file makes such a list a very slow-loading page.
The final join between the listing of the PTR's in the overall PTRList.txt file will
dictate which PTR's make the final cut. The counts can be restored later.

The initial step is to extract all the multi-address PTR's after concatenating the
outputs of the nmap scripts that queried the 50,000 CIDR/16 blocks extracted from
the Current Visitor data. Those scripts are already written & tested. Next, join the
PTRList.txt file (no need for that superfluous Column $3) with the Recent Visitor
data to discover which (of the multi-address PTR's resolved by the nmap scans) are found
in the gratuitously looked up hostnames cataloged by the Webalizer analyses of the
reporting domains. Those many Internet addresses can be readily checked with dig -x
or nslookup; most of the ones that haven't been defensively changed already on the
servers should resolve to their hostnames as listed in the Recent Visitor data. The last
step is to publish these correlations to facilitate the defense of domains undergoing
attack by hosts that cannot be resolved, except by tedious perusing of search engine
data, and therefore are unblockable. Uploading the text files listing the known
addresses of the reported multi-address PTR records will be a slow process, but the
separate listings linked in the presentation webpage will be more accessible on a
one-file-at-a-time basis than an all-encompassing master list that might quickly
become obsolete.

George Langford

With both apologies and thanks to Magic Banana, yesterday I set about to find an independent solution
within which I can follow the logic.

Here is the series of scripts that accomplish the stated task. They are based on the previously
attached PTRListSort.txt and IPv4.May2020.37.nMapoG.txt as illustrative data.

The first script makes the 42 files that will hold the multi-address PTR's and their IPv4 addresses:
awk '{print $1,$2}' 'PTRList.txt' | awk '{print "touch CountsFiles/"$1"."$2".txt ;"}' '-' > Script.MakeFiles.txt

The second script makes 42 additional files, each one ready to contain just one PTR:
awk '{print $1,$2}' 'PTRList.txt' | awk '{print "touch PTR-files/"$1".txt ; "}' > Script.MakePTR-files.txt

The third script writes the PTR names into their just-created files:
awk '{print "echo "$1}' 'PTRList.txt' > Temp0718A.txt ;
awk '{print " > PTR-files/"$1".txt ;"}' 'PTRList.txt' > Temp0718B.txt ;
paste -d ' ' Temp0718A.txt Temp0718B.txt > Script.Fill.PTR-files.txt ; rm Temp0718A.txt Temp0718B.txt

The fourth script creates and lists 42 individual scripts, each one of which collects and writes the joined
address data to its just created file. Note that the first file in the join command contains only one line:
awk '{print "join -a 1 <(sort PTR-files/"$1".txt) <(sort IPv4.May2020.37.nMapoG.txt) | sort -Vk 2 >"}' 'PTRList.txt' > Temp0719A.txt;
awk '{print "CountsFiles/"$1"."$2".txt ;"}' 'PTRList.txt' > Temp0719B.txt;
paste -d ' ' Temp0719A.txt Temp0719B.txt > Script.Fill.CountsFiles.txt; rm Temp0719A.txt Temp0719B.txt

The fifth script makes all the above scripts executable
(the txt extension could be bin, run, sh (Magic Banana's preference)... the terminal doesn't care):
sudo chmod +x Script.MakeFiles.txt; chmod +x Script.MakePTR-files.txt; chmod +x Script.Fill.PTR-files.txt; chmod +x Script.Fill.CountsFiles.txt

The last script executes all the above scripts in one go, one after the other;
don't try this at home until you're exactly sure what's about to happen:
./Script.MakeFiles.txt; ./Script.MakePTR-files.txt; ./Script.Fill.PTR-files.txt; ./Script.Fill.CountsFiles.txt

George Langford

Magic Banana added responses to questions he didn't know I'd asked; our notes apparently crossed in the ether !

My inefficient (but clear to me) scripts needed that extra collection of single-line PTR's because I use them
in my one-at-a-time join command; afterwards, they're not needed any more, but take up a lot of inefficiently
used disk unit storage.

I've fixed the duplicates issue in my join command.

Those unwieldy tables of PTR's and their multiple addresses will soon be scrolling down the screen so far
that the filename may disappear from view; and the -Vk 2 option puts the IPv4's in visually searchable order.

2GB/60kB times 0.3 seconds is seriously longer than times 0.01 second: 2.8 hours vs. 5.6 minutes. Touche.

Magic Banana condensed his already slender script to the utmost:
Now, if you want the duplicates [No !], if you insist on the file name[s?] you chose and if you really want the
repeated PTR in a first field (what only looks like a waste of disk space), it is trivial to adapt my solution:
awk 'FILENAME == ARGV[1] { a[$1] = $2 } FILENAME == ARGV[2] && $1 in a { print $1, $2 >> "CountsFiles/" $1 "." a[$1] ".txt" }' PTRList.txt IPv4.May2020.37.nMapoG.txt

It works ! Thanks for your script's astounding brevity, for your accurate scripting, and for being so diligent.

In the final reckoning, IPv4.May2020.37.nMapoG.txt (60kB) becomes IPv4.May2020.100.nMapoG.txt (2GB).
PTRList.txt can keep the same name.

George Langford