Using External Input Data in Off-the-shelf Web Scrapers

There is a question I’ve wanted to shed some light upon for a long time already: “What if I need to scrape several URL’s based on data in some external database?“.

For example, recently one of our visitors asked a very good question (thanks, Ed):

    “I have a large list of amazon.com asin. I would like to scrape 10 or so fields for each asin. Is there any web scraping software available that can read each asin from a database and form the destination url to be scraped like http://www.amazon.com/gp/product/{asin} and scrape the data?”

This question impelled me to investigate this matter. I contacted several web scraper developers, and they kindly provided me with detailed answers that allowed me to bring the following summary to your attention:
Visual Web Ripper

An input data source can be used to provide a list of input values to a data extraction project. A data extraction project will be run once for each row of input values. You can find the additional information here.
Web Content Extractor

You can use the -at”filename” command line option to add new URLs from TXT or CSV file:

    WCExtractor.exe projectfile -at”filename” -s

projectfile: the file name of the project (*.wcepr) to open.
filename – the file name of the CSV or TXT file that contains URLs separated by newlines.
-s – starts the extraction process

You can find some options and examples here.
Mozenda

Since Mozenda is cloud-based, the external data needs to be loaded up into the user’s Mozenda account. That data can then be easily used as part of the data extracting process. You can construct URLs, search for strings that match your inputs, or carry through several data fields from an input collection and add data to it as part of your output. The easiest way to get input data from an external source is to use the API to populate data into a Mozenda collection (in the user’s account). You can also input data in the Mozenda web console by importing a .csv file or importing one through our agent building tool.

Once the data is loaded into the cloud, you simply initiate building a Mozenda web agent and refer to that Data list. By using the Load page action and the variable from the inputs, you can construct a URL like http://www.amazon.com/gp/product/%asin%.
Helium Scraper

Here is a video showing how to do this with Helium Scraper:


The video shows how to use the input data as URLs and as search terms. There are many other ways you could use this data, way too many to fit in a video. Also, if you know SQL, you could run a query to get the data directly from an external MS Access database like
SELECT * FROM [MyTable] IN "C:\MyDatabase.mdb"

Note that the database needs to be a “.mdb” file.
WebSundew Data Extractor
Basically this allows using input data from external data sources. This may be CSV, Excel file or a Database (MySQL, MSSQL, etc). Here you can see how to do this in the case of an external file, but you can do it with a database in a similar way (you just need to write an SQL script that returns the necessary data).
In addition to passing URLs from the external sources you can pass other input parameters as well (input fields, for example).
Screen Scraper

Screen Scraper is really designed to be interoperable with all sorts of databases. We have composed a separate article where you can find a tutorial and a sample project about scraping Amazon products based on a list of their ASINs.


Source: http://extract-web-data.com/using-external-input-data-in-off-the-shelf-web-scrapers/

Selenium IDE and Web Scraping

Selenium is a browser automation framework that includes IDE, Remote Control server and bindings of various flavors including Java, .Net, Ruby, Python and other. In this post we touch on the basic structure of the framework and its application to  Web Scraping.
What is Selenium IDE


Selenium IDE is an integrated development environment for Selenium scripts. It is implemented as a Firefox plugin, and it allows recording browsers’ interactions in order to edit them. This works well for software tests, composing and debugging. The Selenium Remote Control is a server specific for a particular environment; it causes custom scripts to be implemented for controlled browsers. Selenium deploys on Windows, Linux, and iOS. How various Selenium components are supported with major browsers read here.
What does Selenium do and Web Scraping

Basically Selenium automates browsers. This ability is no doubt to be applied to web scraping. Since browsers (and Selenium) support JavaScript, jQuery and other methods working with dynamic content why not use this mix for benefit in web scraping, rather than to try to catch Ajax events with plain code? The second reason for this kind of scrape automation is browser-fasion data access (though today this is emulated with most libraries).

Yes, Selenium works to automate browsers, but how to control Selenium from a custom script to automate a browser for web scraping? There are Selenium PHP and other language libraries (bindings) providing for scripts to call and use Selenium. It is possible to write Selenium clients (using the libraries) in almost any language we prefer, for example Perl, Python, Java, PHP etc. Those libraries (API), along with a server, the Java written server that invokes browsers for actions, constitute the Selenum RC (Remote Control). Remote Control automatically loads the Selenium Core into the browser to control it. For more details in Selenium components refer to here.


A tough scrape task for programmer

“…cURL is good, but it is very basic.  I need to handle everything manually; I am creating HTTP requests by hand.
This gets difficult – I need to do a lot of work to make sure that the requests that I send are exactly the same as the requests that a browser would
send, both for my sake and for the website’s sake. (For my sake
because I want to get the right data, and for the website’s sake
because I don’t want to cause error messages or other problems on their site because I sent a bad request that messed with their web application).  And if there is any important javascript, I need to imitate it with PHP.
It would be a great benefit to me to be able to control a browser like Firefox with my code. It would solve all my problems regarding the emulation of a real browser…
it seems that Selenium will allow me to do this…” -Ryan S

Yes, that’s what we will consider below.
Scrape with Selenium

In order to create scripts that interact with the Selenium Server (Selenium RC, Selenium Remote Webdriver) or create local Selenium WebDriver script, there is the need to make use of language-specific client drivers (also called Formatters, they are included in the selenium-ide-1.10.0.xpi package). The Selenium servers, drivers and bindings are available at Selenium download page.
The basic recipe for scrape with Selenium:

    Use Chrome or Firefox browsers
    Get Firebug or Chrome Dev Tools (Cntl+Shift+I) in action.
    Install requirements (Remote control or WebDriver, libraries and other)
    Selenium IDE : Record a ‘test’ run thru a site, adding some assertions.
    Export as a Python (other language) script.
    Edit it (loops, data extraction, db input/output)
    Run script for the Remote Control

The short intro Slides for the scraping of tough websites with Python & Selenium are here (as Google Docs slides) and here (Slide Share).
Selenium components for Firefox installation guide

For how to install the Selenium IDE to Firefox see  here starting at slide 21. The Selenium Core and Remote Control installation instructions are there too.
Extracting for dynamic content using jQuery/JavaScript with Selenium

One programmer is doing a similar thing …

1. launch a selenium RC (remote control) server
2. load a page
3. inject the jQuery script
4. select the interested contents using jQuery/JavaScript
5. send back to the PHP client using JSON.

He particularly finds it quite easy and convenient to use jQuery for
screen scraping, rather than using PHP/XPath.
Conclusion

The Selenium IDE is the popular tool for browser automation, mostly for its software testing application, yet also in that Web Scraping techniques for tough dynamic websites may be implemented with IDE along with the Selenium Remote Control server. These are the basic steps for it:

    Record the ‘test‘ browser behavior in IDE and export it as the custom programming language script
    Formatted language script runs on the Remote Control server that forces browser to send HTTP requests and then script catches the Ajax powered responses to extract content.

Selenium based Web Scraping is an easy task for small scale projects, but it consumes a lot of memory resources, since for each request it will launch a new browser instance.



Source: http://extract-web-data.com/selenium-ide-and-web-scraping/

Elevate Your Business With Data Entry Services

The sole aim of many organizations is to progress well in their objectives and hire people who are good and efficient in their work. However, sometimes, there are some work profiles that are mundane in nature but equally important like data entry services. You will be amazed to know that these services also play a crucial role in building the future of an organization.

In fact, with the coming of information technology, the data entry services have actually become a kind of industry, as various businesses need accurate and detailed information for various reasons. Thus they are relying on such services that not only help them in growing but are cost effective too. These data entry services are an asset for any organization irrespective of its size in both the terms of workforce, financial status and area. With the help of such services you are able to get the information on the market trends, your clients and moreover, about the status of your own business. Hence, there is a lot of demand for data entry services in order to do great business.

As you must be aware of the fact that data entry services can be time consuming; hence it requires efficient workforce to execute various tasks perfectly and diligently. Every transaction has to be recorded, processed and analyzed so that the management or the decision-makers can have a clear picture of the actual financial standing of the company. In fact, there are many organizations that are interested in the data of company so that they can strike a business deal with the company in the future; the competitors are also the one's who are constantly following the happenings of the company. However, the most important part that constitutes group are the shareholders, employees, creditors, consumers and the market in general. Therefore, this service plays a significant role in determining the future of the company. Thus, it is taken very seriously by many business enterprises for various reasons that can elevate their businesses by many fractions.

In fact, data entry services are now being outsourced from various leading vendors to further simplify the requirements of every business. Well, these services cover many business activities like document and image processing, data conversion, image enhancement, image editing, catalog processing, and photo manipulation. In fact, you can use data entry services for transferring hard or soft copy to any database format; insurance claims entry; PDF document indexing; online data capture; product catalogs to web based systems; online order entry and follow up; creation of new databases. Moreover, banks, airlines, government agencies, direct marketing services and service providers are using these services for better businesses.

The data services are also utilized for mailing lists; data mining and warehousing; data cleansing; audio transcriptions; legal documents; indexing of vouchers and documents; hand written ballot or card entry; online completion of surveys and responses of customers for various companies. Now its up to the company to whether go for a vendor or hire in-house staff to accomplish tasks in a better way; the main purpose of this service is to offer convenience that can help in curbing time as well as other resources.




Source: http://ezinearticles.com/?Elevate-Your-Business-With-Data-Entry-Services&id=777230

Internet Data Mining - How Does it Help Businesses?

Internet has become an indispensable medium for people to conduct different types of businesses and transactions too. This has given rise to the employment of different internet data mining tools and strategies so that they could better their main purpose of existence on the internet platform and also increase their customer base manifold.

Internet data-mining encompasses various processes of collecting and summarizing different data from various websites or webpage contents or make use of different login procedures so that they could identify various patterns. With the help of internet data-mining it becomes extremely easy to spot a potential competitor, pep up the customer support service on the website and make it more customers oriented.

There are different types of internet data_mining techniques which include content, usage and structure mining. Content mining focuses more on the subject matter that is present on a website which includes the video, audio, images and text. Usage mining focuses on a process where the servers report the aspects accessed by users through the server access logs. This data helps in creating an effective and an efficient website structure. Structure mining focuses on the nature of connection of the websites. This is effective in finding out the similarities between various websites.

Also known as web data_mining, with the aid of the tools and the techniques, one can predict the potential growth in a selective market regarding a specific product. Data gathering has never been so easy and one could make use of a variety of tools to gather data and that too in simpler methods. With the help of the data mining tools, screen scraping, web harvesting and web crawling have become very easy and requisite data can be put readily into a usable style and format. Gathering data from anywhere in the web has become as simple as saying 1-2-3. Internet data-mining tools therefore are effective predictors of the future trends that the business might take.

If you are interested to know something more on Web Data Mining and other details, you are welcome to the Screen Scraping Technology site.




Source: http://ezinearticles.com/?Internet-Data-Mining---How-Does-it-Help-Businesses?&id=3860679

Online Data Entry and Data Mining Services

Data entry job involves transcribing a particular type of data into some other form. It can be either online or offline. The input data may include printed documents like Application forms, survey forms, registration forms, handwritten documents etc.

Data entry process is an inevitable part of the job to any organization. One way or other each organization demands data entry. Data entry skills vary depends upon the nature of the job requirement, in some cases data to be entered from a hard copy formats and in some other cases data to be entered directly into a web portal. Online data entry job generally requires the data to be entered in to any online data base.

For a super market, data associate might be required to enter the goods which have sold in a particular day and the new goods received in a particular day to maintain the stock well in order. Also, by doing this the concerned authorities will get an idea about the sale particulars of each commodity as they requires. In another example, an office the account executive might be required to input the day to day expenses in to the online accounting database in order to keep the account well in order.

The aim of the data mining process is to collect the information from reliable online sources as per the requirement of the customer and convert it to a structured format for the further use. The major source of data mining is any of the internet search engine like Google, Yahoo, Bing, AOL, MSN etc. Many search engines such as Google and Bing provide customized results based on the user's activity history. Based on our keyword search, the search engine lists the details of the websites from where we can gather the details as per our requirement.

Collect the data from the online sources such as Company Name, Contact Person, Profile of the Company, Contact Phone Number of Email ID Etc. are doing for the marketing activities. Once the data is gathered from the online sources into a structured format, the marketing authorities will start their marketing promotions by calling or emailing the concerned persons, which may result to create a new customer. So basically data mining is playing a vital role in today's business expansions. By outsourcing the data entry and its related works, you can save the cost that would be incurred in setting up the necessary infrastructure and employee cost.

E-dataentry is an offshore India based company providing superior quality data mining services to clients across the globe with high level of accuracy at reasonable price.




Source: http://ezinearticles.com/?Online-Data-Entry-and-Data-Mining-Services&id=7713395

How Do We Store Data for Future Data Mining Without Knowing the Future Questions?

Let's talk a little bit about "transparency versus public access" and where it's appropriate, and where it obviously isn't. Not long ago, there was an interesting feature in the TV news, a big to do about nothing, where the First Lady Michelle had traveled to Spain, and as she was on her vacation, she was on vacation as a private citizen. Now whereas, people want transparency, one has to ask where privacy must take precedent, and where transparency should be afforded.

Now, you might not think this is a very good example, but when it comes to online social networks, paparazzi, and privacy all these things are really big issues. Recall when Sarah Palin's yahoo email account was hacked by a college student, Obama supporter in TN? Obviously, that crossed the line, but where do we draw the line online?

Okay so, let's get back to the main question here; How Do We Store Online Data without violating personal property, and how do we protect national security without breaches in data, or violations of personal privacy. And if we anonimize all the data for use at a future time, how should we store it for Future Data Mining Without Knowing the Future Questions?

The information and data could be stored by region, time, frequency, and relevance. It must be stored for a multitude of purposes, and we must determine who may obtain the data, who will use the data, and what will they use it for. You see, there are different ways to store the information categories to be displayed in, or various types of tags to assign it to.

Perhaps, all the information can be stored, every bit of it, and a trusted data inquirer who wants to ask the questions, will have to explain their inquiry to an artificially intelligent computer, and it can act like a Supreme Court review on privacy. In other words, if the reason for the information is not good enough, access to that particular information will be denied. And yes it could use constitutional extrapolations, which would be philosophically based on the same analogy as surgeon seizure rules, or Fifth Amendment rights of self-determination.

As if the data itself would be alive, and the artificial intelligent computer would be the judge deciding if the prosecution would be allowed to ask those questions of the computer data system. In this case you could just store all the information you could possibly take in, and not worry about it. Okay so, that is one option; just store all the data, regardless of what it is. Or another option is to store only some data, data you believe to be important for the future, but knowing the whole truth of the past, is not completely known.

This is problematic however due to "selective prosecution" challenges. You see, one of my biggest fears would be information taken at a context, and used to condemn people or character assassinate them, or incriminate them at a trial, or in the mass media in court of public opinion using stored data, using a computer forensic chain of data, selectively gathered.

We know that the media uses this trick early and often, and they do so in often ruining people's lives. We need to be careful with that. It's serious issue. The reality is you cannot trust humans, they have proven throughout history to be a trustworthy, and you don't have to go very far to find inherent corruptness and individuals of the human species. This being my primary reason for suggesting an AI computer system.

The other concept might be to not collect the data at all, because you don't really need the data, and if you have the data available, we all know that it will be abused. Of course, the proof of innocence could also very well be in that same data, you see that point? But, the chances for abuse is far too great when humans are involved. We've had previous Presidential Administrations use IRS data to attack their enemies, and use the FBI to track political opponents. State Governors have used state police to track persons whom they've had disputes with or political adversaries as well. The abuse of power is quite common.

So, under the opposite model, you could say; No Data from Anyone, Agency, Corporation, or Organization maybe collected period; you can't collect it, you can't have it, and you can't use it. That means you can't use it for good or for evil. Some might say that would be unfortunate because a lot of that data can help prevent crimes, it can help better solve the challenges and problems of our society, and it can help artificial intelligence make the best decisions based on the best information.

If we continually make decisions based on lack of information, is this really a smart way to do planning? If on the other hand we have irrelevant information, bad information, or information taken out of context, we will never be able to make any decisions without very unfortunate unintended consequences, which is what is happening now it seems.

At our think tank we talk a lot about this, but we don't do political correctness, and we aren't about to give the human species a free pass on integrity, they don't deserve it, they haven't earned it, and we all know they cannot be trusted.




Source: http://ezinearticles.com/?How-Do-We-Store-Data-for-Future-Data-Mining-Without-Knowing-the-Future-Questions?&id=4867341

What You Should Know About Data Mining

Often called data or knowledge discovery, data mining is the process of analyzing data from various perspectives and summarizing it into useful information to help beef up revenue or cut costs. Data mining software is among the many analytical tools used to analyze data. It allows categorizing of data and shows a summary of the relationships identified. From a technical perspective, it is finding patterns or correlations among fields in large relational databases. Find out how data mining works and its innovations, what technological infrastructures are needed, and what tools like phone number validation can do.

Data mining may be a relatively new term, but it uses old technology. For instance, companies have made use of computers to sift through supermarket scanner data - volumes of them - and analyze years' worth of market research. These kinds of analyses help define the frequency of customer shopping, how many items are usually bought, and other information that will help the establishment increase revenue. These days, however, what makes this easy and more cost-effective are disk storage, statistical software, and computer processing power.

Data mining is mainly used by companies who want to maintain a strong customer focus, whether they're engaged in retail, finance, marketing, or communications. It enables companies to determine the different relationships among varying factors, including staffing, pricing, product positioning, market competition, and social demographics.

Data mining software, for example, vary in types: statistical, machine learning, and neural networks. It seeks any of the four types of relationships: classes (stored data is used for locating data in predetermined groups), clusters (data are grouped according to logical relationships or consumer preferences), associations (data is mined to identify associations), and sequential patterns (data is mined to estimate behavioral trends and patterns). There are different levels of analysis, including artificial neural networks, genetic algorithms, decision trees, nearest neighbor method, rule induction, and data visualization.

In today's world, data mining applications are available on all size systems from client/server, mainframe, and PC platforms. When it comes to enterprise-wide applications, the size usually ranges from 10 gigabytes to more than 11 terabytes. The two important technological drivers are the size of the database and query complexity. A more powerful system is required with more data being processed and maintained, and with more complex and greater queries.

Programmable XML web services like phone number validation will assist your company in improving the quality of your data needed for data mining. Used to validate phone numbers, a phone number validation service allows you to improve the quality of your contact database by eliminating invalid telephone numbers at the point of entry. Upon verification, phone number and other customer information can work wonders for your business and its constant improvement.




Source: http://ezinearticles.com/?What-You-Should-Know-About-Data-Mining&id=6916646