Python Internet Access using Urllib.Request and urlopen()

The sections below explain what urllib is, how to open a URL and read its HTML, and how to handle errors, download files, and choose between urllib and the requests library.

What is urllib?

urllib is a Python module that can be used for opening URLs. It defines functions and classes to help in URL actions.

With Python, you can also access and retrieve data from the internet, such as XML, HTML, and JSON, and then work with this data directly. In the sections below, we will see how to retrieve data from the web. For example, here we use a Guru99 video URL, access it using Python, and print the HTML file of this URL.

The urllib package groups several modules: urllib.request for opening URLs, urllib.error for the exceptions that requests can raise, urllib.parse for building and parsing URLs, and urllib.robotparser for reading robots.txt files.

How to Open URL using Urllib

Before we run the code to connect to internet data, we need to import the URL library module, or “urllib”.

• Import urllib
• Define your main function
• Declare the variable webUrl
• Then call the urlopen function on the urllib library
• The URL we are opening is the Guru99 tutorial on YouTube
• Next, we print the result code
• The result code is retrieved by calling the getcode function on the webUrl variable we have created
• We convert that to a string, so that it can be concatenated with our string “result code”
• This will be a regular HTTP code “200”, indicating the HTTP request is processed successfully

How to Get an HTML File from a URL in Python

You can also read the HTML file by using the read() function in Python. When you run the code, the HTML file will appear in the console.

• Call the read() function on the webUrl variable
• The read() method allows you to read the contents of data files
• Read the entire content of the URL into a variable called data
• Run the code and it will print the data in HTML format

Here is the complete code:

Python 2 Example

#  
# read the data from the URL and print it
#
import urllib2

def main():
# open a connection to a URL using urllib2
   webUrl = urllib2.urlopen("https://www.youtube.com/user/guru99com")
  
#get the result code and print it
   print "result code: " + str(webUrl.getcode()) 
  
# read the data from the URL and print it
   data = webUrl.read()
   print data
 
if __name__ == "__main__":
  main()

Python 3 Example

#
# read the data from the URL and print it
#
import urllib.request
# open a connection to a URL using urllib
webUrl  = urllib.request.urlopen('https://www.youtube.com/user/guru99com')

#get the result code and print it
print ("result code: " + str(webUrl.getcode()))

# read the data from the URL and print it
data = webUrl.read()
print (data)

How to Handle urllib Errors: URLError and HTTPError

A request does not always succeed. When something goes wrong, urllib raises an exception instead of returning data, so it is important to handle these errors. The urllib.error module defines two main exception classes.

• HTTPError is raised when the server returns an error status code, such as 404 Not Found or 500 Internal Server Error. It carries the status code and the response body.
• URLError is raised when the server cannot be reached at all, for example because of a wrong domain name or no internet connection. It carries a reason attribute that explains the failure.

Because HTTPError is a subclass of URLError, always catch HTTPError first so that each error type is handled separately:

from urllib.request import urlopen
from urllib.error import HTTPError, URLError

try:
    response = urlopen("https://www.guru99.com/no-such-page.html")
    data = response.read()
except HTTPError as error:
    print("HTTP error:", error.code)
except URLError as error:
    print("URL error:", error.reason)

Handling these exceptions keeps your program from crashing and lets you log the problem, retry the request, or fall back to cached data.

How to Download a File from a URL Using urllib

Reading a URL into memory works for small pages, but for images, PDF documents, or datasets it is easier to save the response straight to disk. The urllib.request module offers two simple ways to do this.

The urlretrieve() function downloads a URL directly to a local file in a single line:

from urllib.request import urlretrieve

urlretrieve("https://www.python.org/static/img/python-logo.png", "logo.png")

For more control, open the URL and write the bytes yourself. Open the local file in binary mode because urlopen() returns bytes, not text:

from urllib.request import urlopen

with urlopen("https://example.com/data.zip") as response:
    with open("data.zip", "wb") as file:
        file.write(response.read())

For very large files, read and write in a loop using response.read(8192) so the whole file is never held in memory at once.

urllib vs. the requests Library in Python

urllib is part of the Python standard library, so it works everywhere without installation. The third-party requests library is not built in, but many developers prefer it for everyday HTTP work because its syntax is shorter and more readable.

The main differences are:

• Installation: urllib ships with Python, while requests must be installed with pip.
• Decoding: urllib returns raw bytes that you decode yourself, while requests decodes text and JSON automatically.
• Convenience: requests handles sessions, cookies, and custom headers with less code.
• Dependencies: urllib adds none, while requests relies on urllib3 underneath.

Choose urllib when you want zero dependencies or only need a quick, one-off request. Choose requests for larger projects that involve authentication, sessions, or frequent API calls. Both send the same underlying HTTP requests, so the decision comes down to convenience versus a smaller dependency list.

FAQs