Thingiverse research impossible due to API rate limiting

kuvakaappaus-2017-01-02-11-16-58

I was thinking of re-creating my previous research Cultures of sharing in 3D printing: what can we learn from the licence choices of Thingiverse users? about Thingiverse community.

This article contributes to the discussion by analysing how users of the leading online 3D printing design repository Thingiverse manage their intellectual property (IP). 3D printing represents a fruitful case study for exploring the relationship between IP norms and practitioner culture. Although additive manufacturing technology has existed for decades, 3D printing is on the cusp of a breakout into the technological mainstream – hardware prices are falling; designs are circulating widely; consumer-friendly platforms are multiplying; and technological literacy is rising. Analysing metadata from more than 68,000 Thingiverse design files collected from the site, we examine the licensing choices made by users and explore the way this shapes the sharing practices of the site’s users. We also consider how these choices and practices connect with wider attitudes towards sharing and intellectual property in 3D printing communities.

Impossible to do research – alone

Years have passed and now Thingiverse.com has over 2 000 000 Things. They have API, but it has rate limiting of 300 calls per 5 minute window. I took a look at the API and managed to take it into use. Getting basic info is one API call (naturally). But to get thorough set of information for each Thing, I need to make 7 API calls since details of each Thing are behind distinct API calls. All API requests require authentication. Now lets do some calculations.

  • around 2 million Things
  • for each we need 7 API calls. In total it’s around 14 million API calls.
  • rate limiting is 300 API calls / 5 mins (60 calls / 1 minute):
    • 14 000 000 API calls / 60 calls per minute / 60 minutes / 24 hours
  • It will be 233333 minutes just to make API calls (network latency and everything still missing from the numbers).
  • It will be 3888 hours… 162 days just to collect data.

Ok, that’s not going to work. We have three options:

  1. get better deal for the API from Thingiverse.com
  2. distribute the data collection among researchers.
  3. settle for snapshot and take for example every 10th Thing and collect metadata just for those.
Mainokset

Kommentoi

Täytä tietosi alle tai klikkaa kuvaketta kirjautuaksesi sisään:

WordPress.com-logo

Olet kommentoimassa WordPress.com -tilin nimissä. Log Out /  Muuta )

Google+ photo

Olet kommentoimassa Google+ -tilin nimissä. Log Out /  Muuta )

Twitter-kuva

Olet kommentoimassa Twitter -tilin nimissä. Log Out /  Muuta )

Facebook-kuva

Olet kommentoimassa Facebook -tilin nimissä. Log Out /  Muuta )

Muodostetaan yhteyttä palveluun %s