Accessing Remote Resources

Web pages and data

I have mentioned before how one can access data files on your hard drive, but Python also allows you to access remote data, for example on the internet using the HTTP protocol. This is the protocol with which web servers and web browsers exchange information. The easiest way to do this is to use the requests module. To start off, you just can get the URL:

In [1]:
import requests

response = requests.get('http://xkcd.com/353/')

response holds the response now. You can access the content as text via the text-property:

In [2]:
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
In [3]:
print(response.text[:300])  # only print the first 300 characters
<!DOCTYPE html>
<html>
<head>
<script>
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,documen

You can either just use this information directly, or in some cases you might want to write it to a file. Let's download one of the full resolution files for the Ice coverage data from Problem Set 2:

In [4]:
r2 = requests.get('http://mpia.de/~robitaille/share/ice_data/20060313.npy')
In [5]:
r2.headers
requests.get?
In [6]:
r2.text[:200]
Out[6]:
"\x93NUMPY\x01\x00F\x00{'descr': '>f4', 'fortran_order': False, 'shape': (1100, 1000), }    \n\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@å ;ÿÿÿÿAÂPRB;a\x9dÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ"

However, this doesn't seem to be actual text. Instead, its a binary format. The binary data of the response can be accessed via

In [7]:
r2.content[:200]
Out[7]:
b"\x93NUMPY\x01\x00F\x00{'descr': '>f4', 'fortran_order': False, 'shape': (1100, 1000), }    \n\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@\xe5 ;\xff\xff\xff\xffA\xc2PRB;a\x9d\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff"

Note the little b at the beginning indicating a binary byte-string.

Now we can open a new (binary) file and download the data to the file.

In [8]:
with open('20060313.npy', 'wb') as f:
    f.write(r2.content)

Let's now load and plot the data:

In [9]:
import numpy as np
data = np.load('20060313.npy')
In [10]:
%matplotlib inline
import matplotlib.pyplot as plt
plt.figure(figsize=(12,12))
plt.imshow(data, origin='lower')
Out[10]:
<matplotlib.image.AxesImage at 0x7f55d76beac8>

APIs

Imagine that you want to access some data online. A number of websites now offer an "Application programming interface" (or API) which is basically a way of exchanging information in a machine-readable way.

Above, we used the HTTP protocol which is used for accessing web resources. On top of the protocol sits the API that defines how information is to be exchanged for a particular service. E.g., Google describes in detail its API for making search requests to the Google web servers. You may can imagine that the definition of an API can be an extensive document (and it typically is). Moreover, the data formats have to be specified, e.g., if images are exchanged.

An example of a web service with a particularly simple API is http://api.open-notify.org which we will use in a moment. We will make a request to a service at this site that provides the current location of the International Space Station (ISS).

In [11]:
import requests
response = requests.get("http://api.open-notify.org/iss-now.json")
print(response.status_code) # print the status code of the response.
200

Status codes of "get" requests

  • 200: everything went okay, and the result has been returned (if any).
  • 301: the server is redirecting you to a different endpoint. This can happen when a company switches domain names, or an endpoint name is changed.
  • 401: the server thinks you are not authenticated. This happens when you do not send the right credentials to access an API.
  • 400: the server thinks you made a bad request. This can happen when you do not send along the right data, among other things.
  • 403: the resource you are trying to access is forbidden – you do not have the right permissions to see it.
  • 404: the resource you tried to access was not found on the server.

As we have seen before data of the request are stored in the .text attribute:

In [12]:
print(response.text)
print(type(response.text))
{"iss_position": {"longitude": "78.4496", "latitude": "49.2177"}, "timestamp": 1487328598, "message": "success"}
<class 'str'>

This looks pretty simple, like a Python dictionary. However, it is a string (in fact in JSON "JavaScript Object Notation") that we can easily convert to a dictionary:

In [17]:
response = requests.get("http://api.open-notify.org/iss-now.json")

d = eval(response.text) # eval() executes a string as if it is a
                        # Python command
for i in d:
    print(i,'   ',d[i])
timestamp     1487329028
message     success
iss_position     {'latitude': '35.0567', 'longitude': '109.5185'}

Exercise 1

The service http://api.open-notify.org/astros.json provides information of people who are currently in space. Access the service with Python and print a list of space-people and their space-craft(s).

In [ ]:
   # your solution here

Another Example (Linux only)

Applications on the Linux desktop communicate with each other using something called the dbus. Python provides a module that allows us to use the dbus protocol to communicate with applications, here with the planetarium program kstars. For this kstars must be installed and running on your desktop.

In [19]:
import dbus
bus = dbus.SessionBus() # establish connection to dbus demon
remote_object = bus.get_object("org.kde.kstars", "/KStars")

# org.kde.kstars: application name
# /KStars: object path, since application might export many objects
 
# remote_object.Introspect() # get info on available methods

# generates a namespace
iface = dbus.Interface(remote_object, 'org.kde.kstars') 

# direct the application to center the star 'Vega'
iface.lookTowards('Polaris')

Here a more complex request accessing the data base of Messier objects stored in kstars. Since the response is in XML format there is a helper function extrating one particular XML-tag.

In [20]:
# helper function
def getxmltag(xstring, name):
    d = xstring.rsplit('<'+name+'>')
    d = d[1].rsplit('</'+name+'>')
    return d[0]

# request data for all 110 Messier objects
for i in range(110):
    messier = 'M ' + str(i+1)
    a = iface.getObjectDataXML(messier)
    print(messier, ' : ', getxmltag(a, 'Constellation'), ' : ', 
          getxmltag(a,'Type'), ' : ', getxmltag(a,'Magnitude'))
M 1  :  Taurus  :  Supernova Remnant  :  8.4
M 2  :  Aquarius  :  Globular Cluster  :  1e+02
M 3  :  Canes Venatici  :  Globular Cluster  :  1e+02
M 4  :  Scorpius  :  Globular Cluster  :  1e+02
M 5  :  Serpens Caput  :  Globular Cluster  :  1e+02
M 6  :  Scorpius  :  Open Cluster  :  1e+02
M 7  :  Scorpius  :  Open Cluster  :  1e+02
M 8  :  Sagittarius  :  Gaseous Nebula  :  5
M 9  :  Ophiuchus  :  Globular Cluster  :  1e+02
M 10  :  Ophiuchus  :  Globular Cluster  :  1e+02
M 11  :  Scutum  :  Open Cluster  :  1e+02
M 12  :  Ophiuchus  :  Globular Cluster  :  1e+02
M 13  :  Hercules  :  Globular Cluster  :  1e+02
M 14  :  Ophiuchus  :  Globular Cluster  :  1e+02
M 15  :  Pegasus  :  Globular Cluster  :  1e+02
M 16  :  Serpens Cauda  :  Open Cluster  :  1e+02
M 17  :  Sagittarius  :  Gaseous Nebula  :  6
M 18  :  Sagittarius  :  Open Cluster  :  1e+02
M 19  :  Ophiuchus  :  Globular Cluster  :  1e+02
M 20  :  Sagittarius  :  Gaseous Nebula  :  6.3
M 21  :  Sagittarius  :  Open Cluster  :  1e+02
M 22  :  Sagittarius  :  Globular Cluster  :  1e+02
M 23  :  Sagittarius  :  Open Cluster  :  1e+02
M 24  :  Sagittarius  :  Open Cluster  :  4.6
M 25  :  Sagittarius  :  Open Cluster  :  1e+02
M 26  :  Scutum  :  Open Cluster  :  1e+02
M 27  :  Vulpecula  :  Planetary Nebula  :  7.6
M 28  :  Sagittarius  :  Globular Cluster  :  1e+02
M 29  :  Cygnus  :  Open Cluster  :  1e+02
M 30  :  Capricornus  :  Globular Cluster  :  1e+02
M 31  :  Andromeda  :  Galaxy  :  4.3
M 32  :  Andromeda  :  Galaxy  :  9.1
M 33  :  Triangulum  :  Galaxy  :  6.2
M 34  :  Perseus  :  Open Cluster  :  1e+02
M 35  :  Gemini  :  Open Cluster  :  1e+02
M 36  :  Auriga  :  Open Cluster  :  1e+02
M 37  :  Auriga  :  Open Cluster  :  1e+02
M 38  :  Auriga  :  Open Cluster  :  1e+02
M 39  :  Cygnus  :  Open Cluster  :  1e+02
M 40  :  Ursa Major  :  Catalog Star  :  8.4
M 41  :  Canis Major  :  Open Cluster  :  1e+02
M 42  :  Orion  :  Gaseous Nebula  :  4
M 43  :  Orion  :  Gaseous Nebula  :  9
M 44  :  Cancer  :  Open Cluster  :  1e+02
M 45  :  Taurus  :  Open Cluster  :  1.6
M 46  :  Puppis  :  Open Cluster  :  1e+02
M 47  :  Puppis  :  Open Cluster  :  1e+02
M 48  :  Hydra  :  Open Cluster  :  1e+02
M 49  :  Virgo  :  Galaxy  :  9.3
M 50  :  Monoceros  :  Open Cluster  :  1e+02
M 51  :  Canes Venatici  :  Galaxy  :  8.9
M 52  :  Cassiopeia  :  Open Cluster  :  1e+02
M 53  :  Coma Berenices  :  Globular Cluster  :  1e+02
M 54  :  Sagittarius  :  Globular Cluster  :  1e+02
M 55  :  Sagittarius  :  Globular Cluster  :  1e+02
M 56  :  Lyra  :  Globular Cluster  :  1e+02
M 57  :  Lyra  :  Planetary Nebula  :  9.7
M 58  :  Virgo  :  Galaxy  :  10
M 59  :  Virgo  :  Galaxy  :  11
M 60  :  Virgo  :  Galaxy  :  9.8
M 61  :  Virgo  :  Galaxy  :  10
M 62  :  Ophiuchus  :  Globular Cluster  :  1e+02
M 63  :  Canes Venatici  :  Galaxy  :  9.3
M 64  :  Coma Berenices  :  Galaxy  :  9.3
M 65  :  Leo  :  Galaxy  :  10
M 66  :  Leo  :  Galaxy  :  9.7
M 67  :  Cancer  :  Open Cluster  :  1e+02
M 68  :  Hydra  :  Globular Cluster  :  1e+02
M 69  :  Sagittarius  :  Globular Cluster  :  7.6
M 70  :  Sagittarius  :  Globular Cluster  :  1e+02
M 71  :  Sagitta  :  Globular Cluster  :  1e+02
M 72  :  Aquarius  :  Globular Cluster  :  1e+02
M 73  :  Aquarius  :  Open Cluster  :  1e+02
M 74  :  Pisces  :  Galaxy  :  9.8
M 75  :  Sagittarius  :  Globular Cluster  :  1e+02
M 76  :  Perseus  :  Planetary Nebula  :  12
M 77  :  Cetus  :  Galaxy  :  9.7
M 78  :  Orion  :  Gaseous Nebula  :  8
M 79  :  Lepus  :  Globular Cluster  :  1e+02
M 80  :  Scorpius  :  Globular Cluster  :  1e+02
M 81  :  Ursa Major  :  Galaxy  :  7.8
M 82  :  Ursa Major  :  Galaxy  :  9.2
M 83  :  Hydra  :  Galaxy  :  8.2
M 84  :  Virgo  :  Galaxy  :  10
M 85  :  Coma Berenices  :  Galaxy  :  10
M 86  :  Virgo  :  Galaxy  :  9.9
M 87  :  Virgo  :  Galaxy  :  9.6
M 88  :  Coma Berenices  :  Galaxy  :  10
M 89  :  Virgo  :  Galaxy  :  11
M 90  :  Virgo  :  Galaxy  :  10
M 91  :  Coma Berenices  :  Galaxy  :  11
M 92  :  Hercules  :  Globular Cluster  :  1e+02
M 93  :  Puppis  :  Open Cluster  :  1e+02
M 94  :  Canes Venatici  :  Galaxy  :  8.9
M 95  :  Leo  :  Galaxy  :  11
M 96  :  Leo  :  Galaxy  :  10
M 97  :  Ursa Major  :  Planetary Nebula  :  12
M 98  :  Coma Berenices  :  Galaxy  :  11
M 99  :  Coma Berenices  :  Galaxy  :  10
M 100  :  Coma Berenices  :  Galaxy  :  10
M 101  :  Ursa Major  :  Galaxy  :  8.2
M 102  :  Draco  :  Galaxy  :  11
M 103  :  Cassiopeia  :  Open Cluster  :  1e+02
M 104  :  Virgo  :  Galaxy  :  9.2
M 105  :  Leo  :  Galaxy  :  10
M 106  :  Canes Venatici  :  Galaxy  :  9.1
M 107  :  Ophiuchus  :  Globular Cluster  :  1e+02
M 108  :  Ursa Major  :  Galaxy  :  11
M 109  :  Ursa Major  :  Galaxy  :  11
M 110  :  Andromeda  :  Galaxy  :  8.9
In [ ]: