Skip to main content

Scrape the Web with ChatGPT

In the vast ocean of the internet, data is the treasure that everyone seeks. Whether it's for business, research, or education, the ability to efficiently gather and analyze data can be a game-changer. This is where the concept of web scraping comes into play. As shown in the experiment below, Web scraping is a method used to extract data from websites and save it to a local file in your computer or to a database in table (tabular) format. With the advent of ChatGPT's new scraping capability, this process has become even more accessible and efficient.

Web scraping with ChatGPT simplifies the process of retrieving repetitive data from websites. Instead of manually searching and recording data, ChatGPT can automate this process, saving time and reducing the potential for human error. It's like having a digital assistant that can tirelessly gather data with precision, allowing users to focus on interpreting and using that data.

Consider the potential use cases for this technology. A real estate company could use ChatGPT's scraping capability to gather property listings from various websites, compiling them into a comprehensive database for analysis. Similarly, a job seeker could use it to pull postings from multiple job boards, making their search more efficient. In the academic world, a researcher could use it to gather data for a meta-analysis, while a teacher could use it to compile resources for a lesson plan. Even lottery enthusiasts could use it to track winning numbers over time, as demonstrated in a previous example.

The impact of simplified web scraping on business is significant. It can streamline market research, competitor analysis, and customer sentiment analysis, among other things. By automating data collection, businesses can focus more on data analysis and decision-making, potentially leading to more informed strategies and better outcomes. It can also level the playing field for smaller businesses, giving them access to the same wealth of data as their larger counterparts.

In the realm of education, ChatGPT's scraping capability can revolutionize the way research is conducted. Students and educators can gather data more efficiently, leaving more time for analysis and discussion. It can also make project-based learning more accessible, as students can easily gather real-world data for their projects. Furthermore, it can foster digital literacy, teaching students about the importance of data and how to work with it.

In conclusion, the new web scraping capability of ChatGPT is a powerful tool that can transform the way we gather and use data. By automating and simplifying the process, it allows users to focus on what truly matters: turning data into knowledge and action. Whether you're a business looking to gain a competitive edge, an educator aiming to enrich your lessons, or a curious individual seeking to understand the world better, web scraping with ChatGPT opens up a world of possibilities.

This post has been cowritten with ChatGPT. Please beware that AI is fascinating and can catch your full attention for hours. Get the balance right! > more?

The Experiment

The site contains the Swiss Lotto Results. 

Suppose we want to extract these results for statistical purposes (or to train a neural network and try to win in the next draw ;-) - This extraction is straightforward with ChatGPT's Scraper plugin.

The Scraper plugin allows to extract data from websites with a single prompt. It's as simple as that!

(continued with the 10 latest results)

We can then use another prompt to transform the text into the Excel-readable CSV format:

And finally we can apply some transformations directly from a prompt, too:

A click on "copy code" in ChatGPT and some short operations in Excel and we have finished scraping the data from the Website directly to Excel: 

Popular posts from this blog

Rethinking Exams in the Age of Chatbots

Today's experiment explores the potential of ChatGPT to assist me in the realm of exams, particularly within my field of teaching, Business Analysis. Can ChatGPT create a question for an exam? Indeed, it can do even more! In the experiment below, ChatGPT showcases (to varying extents) its ability to not only generate an exam question but also to engage in several related tasks: Generate a difficult critical question Solve the question Critically evaluate the solution (its own solution!) Attribute a mark between 1 and 6 Attribute a severe mark between 1 and 6 Defend a mark Refute a mark Generate a variation of the question which is difficult to solve Generate a variation of the question which is even more difficult to solve Give a correction scale with marks from 1 to 6  The impact of AI and chatbots like ChatGPT impact on university exams is becoming increasingly evident. Students and teachers alike are beginning to discover these tools, questioning their potential to generate, cor

Harnessing AI to Refine Educational Courses: My Experience with ChatGPT's Capabilities

In a recent interaction, I showcased the potential of OpenAI's ChatGPT in assisting educators to refine their course structures. My experiment began by predefining a context using a comprehensive list of Business Analysis Techniques. Following this preprompt, I presented a current undergraduate course structure that needed modification. The goal? To identify and recommend new Business Analysis techniques appropriate for each chapter of the course.   The Power of Contextual Communication with ChatGPT One of the most striking elements in this experiment was the emphasis on context. My initial "preprompt" set the stage by providing ChatGPT with a menu of options to consider. This allows for a directed, topic-specific conversation, where the AI can pull from a given set of data. The "postprompt" then elaborated on the current course structure, allowing ChatGPT to compare, contrast, and make recommendations based on the established context. Quality and Limitations of

ChatGPT predicts Real Estate Prices !

Is it possible to predict real estate rental prices without writing a single line of code? I asked ChatGPT to generate a complete solution. And was stunnded by its answers... Artificial Intelligence has once again pushed the boundaries of what we thought was possible. The AI field has presented us with a stunning demonstration of its ability to not only assist in writing code, but to single-handedly create a complete, working Random Forest Regression solution. Yes, you read that right. ChatGPT, an AI model developed by OpenAI, is now proficient enough to autonomously develop a complex web application to predict real estate prices. And the most exciting part? You don’t need to write a single line of code! How Does it Work? In a fascinating experiment (below at the end of the post), ChatGPT was tasked with developing a complete web solution on PythonAnywhere. The brief was complex: create a webpage where users could upload a CSV file with geographical coordinates and real estate rental p