Ask HN: How to Evaluate IP Dataset?

[punkpeye]

I've been researching solutions for cost effective method to enrich millions of IPs per day (only need country, city, ASN), and I came across ip-api.com.

The service is mentioned in a few random Reddit threads as cost effective option, but I cannot find any discussions about accuracy of data.

How would one even go about verifying it?

For context, I am paying at the moment a few thousand/per month to another provider for IP data. I am not really using this data for anything other than have enough data to troubleshoot customer support/fraud. This service is promising unlimited IP data for USD 16/month, which sounds too good to be true, but there are many positive reviews across the Internet, so I am trying to evaluate if it is worth adopting.

[lpellegr]

As some said, don’t trust pricing or marketing claims, run your own benchmark based on your requirements.

Measurement-based probing has been used by major IP intelligence providers for years. Some just market it better than others But it’s not a silver bullet, a lot of IPs (mobile, CGNAT, enterprise) aren’t consistently probeable or yield ambiguous results.

In practice, no single method wins. Hybrid approaches usually perform better.

(For transparency: I work for https://ipregistry.co, worth including in your benchmark if you’re comparing providers.)

[reincoder]

I work for IPinfo. We provide a free country and ASN database on a free tier with an unlimited amount of requests. You can download the entire database or use the API services. For country and ASN, it is free.

However, we do not offer city level data for free. > How would one even go about verifying it?

We believe we are the most accurate IP data provider out there, but you should come to that conclusion yourself.

I can tell you why our data is super accurate compared to the rest of the industry. The industry as a whole uses self-reported information that is offered by ASN and ISPs. It is called "geofeed". The issue with geofeed is that IP geolocation providers do not tend to verify the accuracy. Many providers just aggregate these public records and repeat what the ISPs and ASNs want them to tell them. This is a quite bad practice.

So we built a network of distributed servers (currently 1360 servers across 160 countries) that run ping, traceroute and other internet measurements and try to infer the location of IP geolocations. This means when you come to asking how do I know you are accurate, we can share our active measurement data and tell you that this is the evidence.

Now, comes the qustions of how you identify accuracy yourself.

First, if you have access to a large pool of known locations of IP addresses, you can run comparisons across different vendors. You need a GPS-backed device to locate IP addresses.

If you do not have a large pool of well-known location IPs, you can take a sample of IP addresses and check them yourself across multiple vendors. You can then use a tool like ping.sx or our own tool ipinfo.io/probenet/live to see evidence of where these IP addresses are located based on latency.

Do not bet on consensuses among IP geolocation providers; run your own tests.

Our data was evaluated by peer-reviewed academic research. You can take a look at that as well, if you want.

> I am not really using this data for anything other than have enough data to troubleshoot customer support/fraud.

Now, I will be honest...you should not pay anything to us. The way you have describing your issue, it seems like the free services we already offer that should satisfy your need.

Do you really need large scale IP address enrichment of all the IP addresses that visit your website? If yes, then for the first layer use our free data that provides ASN and country information.

Then, when you need troubleshooting with your customers, you can look up those individual IP addresses for free on our website, where we provide all our data for free access.

---

Let me know if you need any help, always happy to answer questions.

[punkpeye]

So many of my open questions answered in one answer. Thank you.

A follow up based on new information - if 'geofeed' identifies something with wrong geo location, and your method detects different geolocation, what do I see as the consumer consuming your API? I am assuming the inferred data, but that also feels counter-intuitive (since the data does not align with what ASN/ISP are reporting).

How often does your active measurement data disagree with geofeed data?

How do you handle mobile/cellular IPs

> Do you really need large scale IP address enrichment of all the IP addresses that visit your website? If yes, then for the first layer use our free data that provides ASN and country information.

If I am troubleshooting a support case that is days/weeks/months old, wouldn't this mean that enriching this information at a later date may give me different data than what it was associated with at the time the requests were made? My understanding was that IPs get re-assigned.

How frequently do IP-to-location mappings change in practice?

Do you offer historical IP data snapshots?

[reincoder]

> I am assuming the inferred data, but that also feels counter-intuitive (since the data does not align with what ASN/ISP are reporting).

That is a very good question. Now, geofeed does not have a verification system. Active measurement is something we use to verify ASN or ISP itself.

Even active measurement has its own limitations. Now in those case where we see active measurements not producing reliable data, we do reach out to ISPs and ASNs to purchase a server in their facility. Geofeed as a system is voluntary and most major ISPs actually do not maintain or even publish that. For example, today I found out a major UK-based telecom geolocated 500k IP addresses in a town with 200k people. ISPs are not inherently incentivized to maintain the accuracy of their self-reported, voluntarily published location data. So, we do proactive outreach to purchase a server from them so we can provide consistent accurate data for their IP addresses.

On the matter of advertised locations not matching actual location, I highly recommend reading this: https://ipinfo.io/blog/vpn-location-mismatch-report

For residential ISPs, we do a lot of outreach and open communication to build a good partnership with them. The goal is that we pay for the privilege to report accurate data for them.

> How often does your active measurement data disagree with geofeed data?

Very frequently.

Here is the summary peer reviewed research paper on this matter: https://community.ipinfo.io/t/ip-geolocation-and-geofeeds-wh...

Active Measurement (1,330 probes, 27.7M RTTs):

  - Country-level: 92.0% accurate → 8% wrong country
  - City-level: 79.6% accurate → 20.4% wrong city

Mobile Device GPS (169 devices, 24 countries):

  - Country-level: 84.5% accurate
  - City-level: 29.9% accurate → 70% wrong city

> How do you handle mobile/cellular IPs

Primarily through active measurement, we are also running a lot of research around more reliable mobile geolocation data.

Because our data is updated daily, I think due to the refresh rate we have an accuracy advantage.

> If I am troubleshooting a support case that is days/weeks/months old, wouldn't this mean that enriching this information at a later date may give me different data than what it was associated with at the time the requests were made? My understanding was that IPs get re-assigned.

You will be surprised to know that historical IP location does not have much demand.

If you are evaluating a support case after some time, you should work with your current data. If the customer raises a question, you address this in real time with their current IP address.

Usually, I do not recommend storing historic IP geolocation information. In most operations, the enrichment happens in real time within the day. Unless you want to do periodic reporting of some sort.

Internally, we of course have the data, but because our IP geolocation is so accurate, it currently sits at around 700 MB. If you add a historical layer to that data, it will be a terabyte of data. There is not much consumer need for it.

> How frequently do IP-to-location mappings change in practice?

https://ipinfo.io/blog/how-many-ips-change-geolocation-over-...

On the city level is 1.3% each day and 16% each month.

> Do you offer historical IP data snapshots?

I highly recommend that you work with current day's data.

In cases where we provide historical data, it is usually for academic research.

---

Let me know if you have any more questions.

[punkpeye]

> On the matter of advertised locations not matching actual location, I highly recommend reading this: https://ipinfo.io/blog/vpn-location-mismatch-report

Good read

Do you happen to know if anyone is compiling all of this data about VPNs into one place? It would be super interesting to know which VPNs are providing genuine services vs masquerading the locations. Maybe even an SEO for you.

> I highly recommend that you work with current day's data.

Just to clarify: You are suggesting that we don't pro-actively enrich every IP address, store IPs, and only enrich them when troubleshooting something?

[reincoder]

> Do you happen to know if anyone is compiling all of this data about VPNs into one place? It would be super interesting to know which VPNs are providing genuine services vs masquerading the locations. Maybe even an SEO for you.

We made that report independently and, according to our analysis, we only identified three VPNs: Windscribe, Mullvad, and iVPN to not have virtual VPN server locations.

> Just to clarify: You are suggesting that we don't pro-actively enrich every IP address, store IPs, and only enrich them when troubleshooting something?

I think you should experiment with this yourself a little. The Lite API is completely free. So you can do ingestion enrichment and post-enrichment enrichment. See what works best for you.

[punkpeye]

Did a quick dive to explore viability of migrating to ipinfo. My idea was: use lite version for enriching everything and then use pay-as-you-go for enriching authenticated user sessions.

I couldn't get /lite/ to work. In a sample of IPs I've tried with, multiple are returning 404. Your website for the same IPs is returning information. Looks like these are just not included in the lite dataset?

Turns out there is no pay-as-you-go tier. Subscription is the only option. Not a deal breaker, but dissapointing setup.

[reincoder]

> I couldn't get /lite/ to work.

Email me: [e-mail]

I think there is an issue with setting up our API.

[punkpeye]

Just to close the (public) loop, the issue was that we were using wrong API endpoint: ipinfo.io/lite instead of api.ipinfo.io/lite.

Thank you Abdullah

[punkpeye]

I tried using the /lite/ endpoint to get country data, but it is giving 404 errors for valid IPs.

{ "status": 404, "error": { "title": "Wrong ip", "message": "Please provide a valid IP address" } }