'C# Discord Embed Emoji Scraping
So I have set up a program that the goal of which is to run through every possible ID and test that ID to see if there is a discord emoji URL associated with it, if there is no emoji associated with that URL the website will not respond. Example link Below;
https://cdn.discordapp.com/emojis/1000000000000000001.png?v=1&size=100
However if there is an emoji associated with that URL, you will be able to see that emoji by following the link, example below;
https://cdn.discordapp.com/emojis/910272514863824951.png?v=1&size=100
Now I want to scrape all possible URL's for every working discord emoji link. If a working ID is found it is logged to a text file. If there is no response it is mentioned in the console and it keeps testing ID's.
Is there a way to speed this process up? I am still very new to C# and I am sure this could be quicker with better code.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Net;
using System.IO;
using System.Net.Http;
namespace Discord_Emoji_Scraper
{
internal class Program
{
static void Main(string[] args)
{
start:
using (WebClient webClient = new WebClient())
try
{
if (id < 1000000000000000000)
{
id++;
string WordID = "" + id;
string disurl = "https://cdn.discordapp.com/emojis/" + WordID + ".png?v=1&size=100";
if (WordID.Length < 18)
{
int diff = 18 - WordID.Length;
WordID += new string('0', diff);
}
HttpWebRequest request = WebRequest.Create(disurl) as HttpWebRequest;
request.Method = "HEAD";
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
string filePath = @"G:\EmojisPng.txt";
string content = disurl;
using (StreamWriter outputFile = new StreamWriter(filePath))
{
outputFile.WriteLine(content);
Console.Write(id + " Does Work!");
goto start;
}
}
else if (id > 1000000000000000000)
{
Console.WriteLine("Done");
return;
}
}
catch
{
Console.WriteLine(id + " Not Work");
goto start;
}
}
public static long id = 10000000000000000;
}
}
Solution 1:[1]
An issue here is the amount of times you're going to have to loop this, it's going to take a very long time regardless. In this code you're looping 990,000,000,000,000,000 times. If we assume each iteration takes exactly one millisecond to complete, this loop will only take a slight 31,478,937 years to complete. Not really good at this webclient stuff but you're probably gonna have to take a different approach.
TLDR: The current code is going to take 31,478,937 years to scrape every possible ID
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|---|
Solution 1 | BoomCode |