Our mission is to bring truth back to the internet.Read more โ†’

Last updated on September, 14

Fact checked

2021 Rankings

Best Web Data Extractors

sub category illustration
Are you looking for data extraction software that will make your life easier? With web data extractors, you can quickly import, clean, explore and analyze data from any web page. Check out web data extractors now and start scraping data automatically!

Frequently Asked Questions on Web Data Extractors

Can I get a list of websites or pages that are not saved?

No. You must specify the URL for any page before you extract it using a web data extractor. Since it does not capture all the information, some website owners may want to keep their site off-limits from the use by this application.

Can I resume an interrupted session in WDE?

Yes. You can resume an interrupted session in a web data extractor. Use the 'File - Open' menu command to open the previously stopped session's log file.

I set up a project with "URLs from File" extraction, enter the filename - but WDE can not find any link in the file?

Make sure the file exists on disk. The file must have a URL line-by-line, other format is not supported, WDE will accept only lines that start with HTTP:// text. Also, web data extractors will not accept URLs that point to image/binary files because those files will not have any text data to extract.

When I run WDE, it sucks all my computer power; the screen is hardly refreshing. What should I do?

It seems you are using a high number of threads. Decrease the thread value to "5" in the "New Session - Other" tab. WDE can launch multiple threads simultaneously. But remember, too high a thread setting may be too much for your computer and/or internet connection to handle it and also puts an unfair load on the host server, which may slow the process down.

I set up a project with "WebSite" extraction, but no email was found?

Few things may cause this: First, not all website owners put their email addresses on their website/contact page. Some websites use forms in their contact/support pages. Second, check the "Depth" setting. Depth tells WDE how many levels to dig down within the specified website.

Can web data extractor extract map IDs?

Yes. Instructions on how to use web data extractor are on the web data extractor help page. You'll need a lat-long list of coordinates for each point you want to gather information about except for Facebook, which lists the location as a general region in their 'likes.'

What query string should I use for specific searches?

Use the import query string to edit or create a new query. For specific searches, you can also specify a search engine and keywords in these fields. You may get many irrelevant results with general queries without specifying a particular term to search. You can also narrow your results by checking the "Specific Term" box.

How long does it take for my friends to get a friend request on Facebook accepted?

A web data extractor will scrape your Facebook profile, check who has a non-private account visible to you (that hasn't already accepted your friend request), and display all their names in an orderly spreadsheet format. So, you can select which ones to unblock or delete.

What database monitoring tools do you use for your database?

Monitoring tools we use to monitor the database are primarily Nagios. We also make sure to pay close attention to spikes in load 50-90% higher than the average over a 10-minute interval, as this would indicate an issue with data extraction.

What if some of my database columns fill up with data?

You can specify an overflow threshold in the config for your web data extractor. The process which extracts the data from the original URL and imports it into Neos exposes underscores. When you import into Neos, they are changed to spaces so that our assistant on top of SQL on top of MongoDB doesn't break INSERTs anymore. 

This site uses cookies. By continuing to use this website, you agree to their use. Read privacy policy for more info.