Skip to main content

Scrape the Web with ChatGPT

In the vast ocean of the internet, data is the treasure that everyone seeks. Whether it's for business, research, or education, the ability to efficiently gather and analyze data can be a game-changer. This is where the concept of web scraping comes into play. As shown in the experiment below, Web scraping is a method used to extract data from websites and save it to a local file in your computer or to a database in table (tabular) format. With the advent of ChatGPT's new scraping capability, this process has become even more accessible and efficient.


Web scraping with ChatGPT simplifies the process of retrieving repetitive data from websites. Instead of manually searching and recording data, ChatGPT can automate this process, saving time and reducing the potential for human error. It's like having a digital assistant that can tirelessly gather data with precision, allowing users to focus on interpreting and using that data.

Consider the potential use cases for this technology. A real estate company could use ChatGPT's scraping capability to gather property listings from various websites, compiling them into a comprehensive database for analysis. Similarly, a job seeker could use it to pull postings from multiple job boards, making their search more efficient. In the academic world, a researcher could use it to gather data for a meta-analysis, while a teacher could use it to compile resources for a lesson plan. Even lottery enthusiasts could use it to track winning numbers over time, as demonstrated in a previous example.

The impact of simplified web scraping on business is significant. It can streamline market research, competitor analysis, and customer sentiment analysis, among other things. By automating data collection, businesses can focus more on data analysis and decision-making, potentially leading to more informed strategies and better outcomes. It can also level the playing field for smaller businesses, giving them access to the same wealth of data as their larger counterparts.

In the realm of education, ChatGPT's scraping capability can revolutionize the way research is conducted. Students and educators can gather data more efficiently, leaving more time for analysis and discussion. It can also make project-based learning more accessible, as students can easily gather real-world data for their projects. Furthermore, it can foster digital literacy, teaching students about the importance of data and how to work with it.

In conclusion, the new web scraping capability of ChatGPT is a powerful tool that can transform the way we gather and use data. By automating and simplifying the process, it allows users to focus on what truly matters: turning data into knowledge and action. Whether you're a business looking to gain a competitive edge, an educator aiming to enrich your lessons, or a curious individual seeking to understand the world better, web scraping with ChatGPT opens up a world of possibilities.

This post has been cowritten with ChatGPT. Please beware that AI is fascinating and can catch your full attention for hours. Get the balance right! > more?


The Experiment

The site https://www.lotteryextreme.com/swiss-lotto/results contains the Swiss Lotto Results. 


Suppose we want to extract these results for statistical purposes (or to train a neural network and try to win in the next draw ;-) - This extraction is straightforward with ChatGPT's Scraper plugin.



The Scraper plugin allows to extract data from websites with a single prompt. It's as simple as that!


(continued with the 10 latest results)

We can then use another prompt to transform the text into the Excel-readable CSV format:


And finally we can apply some transformations directly from a prompt, too:


A click on "copy code" in ChatGPT and some short operations in Excel and we have finished scraping the data from the Website directly to Excel: 












Popular posts from this blog

An Ethical Framework for Exams and Continuous Assessment with AI

Throughout the history of education, a multitude of diverse and even contentious methodologies have been employed to assess knowledge and aptitude. Several assessment methods have been employed, ranging from oral examinations to written tests, all with the overarching objective of discerning the extent to which students comprehend the subject matter in a manner that is equitable, comprehensive, and optimal. In the present context, amidst the cusp of a forthcoming epoch, the ongoing discourse persists, albeit with a futuristic inclination. The emergence of Artificial Intelligence (AI) has brought about transformative changes in the field of education, revolutionizing the tools and methodologies employed for assessment and evaluation. As we transition into this emerging era, it is crucial to discuss the implications for the fundamental principles and ideals of education. The conventional methods of assessment, commonly characterized by high-stakes examinations and standardized tests, ha...
Read more

Rethinking Exams in the Age of Chatbots

Today's experiment explores the potential of ChatGPT to assist me in the realm of exams, particularly within my field of teaching, Business Analysis. Can ChatGPT create a question for an exam? Indeed, it can do even more! In the experiment below, ChatGPT showcases (to varying extents) its ability to not only generate an exam question but also to engage in several related tasks: Generate a difficult critical question Solve the question Critically evaluate the solution (its own solution!) Attribute a mark between 1 and 6 Attribute a severe mark between 1 and 6 Defend a mark Refute a mark Generate a variation of the question which is difficult to solve Generate a variation of the question which is even more difficult to solve Give a correction scale with marks from 1 to 6  The impact of AI and chatbots like ChatGPT impact on university exams is becoming increasingly evident. Students and teachers alike are beginning to discover these tools, questioning their potential to generate, cor...
Read more

How to augment a Course with AI and Student Personas

As an educator in the field of business analysis, I've always sought innovative ways to enhance the learning experience for my students. My latest experiment (see below) into this endeavor involved leveraging ChatGPT to not only understand the learning process better but also to tailor it to individual student needs. Here's a glimpse into this fascinating journey. It began with identifying eight phases of a learning process with ChatGPT. These phases ranged from orientation to reflection and evaluation, encompassing the entire spectrum of a student's learning journey. The goal was to understand how students navigate through these phases and how they could be supported in each step. Building on these phases, I adapted them into a model for student personas . This involved considering various aspects of a student's academic life, personal characteristics, and learning styles. From this model, I created ten distinct student personas. Each persona represented a unique c...
Read more