[karakal]

I'm curious, how does one gain access to flight schedules/fares? Is this something that anyone can get their hands on and create a service (complexity aside), or do you need some sort of license that costs thousands of dollars?

Does each airline have their own way of exporting this data? Is there a single entity that aggregates from all of them? How does the actual data look like? (Is it a dump every X hours, or something more modern like a stream you can subscribe to?).

[dmbaggett]

Fares and rules come from ATPCO (http://atpco.net). Flights come from OAG (http://www.oag.com). Seat availability requires data directly from the carrier, usually via a live query to the carrier's reservation system.

The GDS companies get this data from these sources and then in turn provide (crappy) APIs for customers to use to query it.

In general, if you're doing bookings you can get small amounts of the data (a query at a time) from a GDS. Otherwise you're looking at millions of dollars per year. And then you need to write code to parse it and price tickets using it (approximately 1M LoC if you're terse about it -- more like 30M if you're a GDS).

(I know all this because I co-founded ITA Software, whose software now powers Google Flights.)

[morgante]

This is what I love about HN. Someone asks a good technical question and literally one of the most qualified people in the world answers it.

[notahacker]

I love the fact that literally one of the most qualified people in the world answers it and my first response is hmmm... hasn't OAG been replaced by Innovata as a schedules provider by now?

[solaarphunk]

Thank you for building ITA Matrix - it is amazing tool and I don't know what I would do without it.

[dmbaggett]

Lots of other people did a lot more of the actual building than I did, but on their behalf, "You're welcome!"

[wfunction]

> Thank you for building ITA Matrix - it is amazing tool and I don't know what I would do without it.

Honest question (I swear I'm not trying to troll): what do you actually do with it that you can't do (or do as well as) with Google Flights or Kayak or some other site like that?

I ask because I've used ITA Matrix and never managed to find a cheaper flight than I can through "normal" means... but I rarely fly multi-destination (and basically never do more complicated stuff) so I'm not sure if the use case is beyond mine or if I just don't know how to utilize it in a useful manner.

[Veratyr]

- Time bars (incredibly handy for optimising for things like tight connections, which can be essential when flying international and missing a connection means being stranded for a day): https://i.imgur.com/RteViUF.png

- Multiple departure/destination airports (you can search SFO,SJC,OAK to LGB,LAX,SNA and back in a single search). To be fair Google supports this on desktop too (but doesn't on mobile).

- Better search for return flights with variable stay lengths (Google only does a 5x5 matrix, ITA Matrix does a full month).

However, perhaps the biggest downside is that you can't actually book through ITA Matrix. It only finds the fare, you have to find it elsewhere to actually buy it (although Hipmunk takes routing codes, which makes it easier)

[chch]

One of the features I like to use a lot is the ability to specify fair classes. As an example, say I want to fly to Tokyo, and I am an Alaska Airlines mileage plan member. Alaska Airlines does not fly to Tokyo, but it has deals with airlines that do. However, sometimes the fare classes are quite complicated. For instance, "Economy" is broken down into many buckets, and not all are created equal. For instance, Delta has at least 13 buckets for Economy, and they each 'fare class' award different amounts of mileage to Alaska flyers [1]:

E: 25% Mileage

L, U, T, X, V: 50% Mileage

H, Q, K: 75%

B, M, S: 100% Mileage

Y: 125% Mileage

If you search on Google Flights, these will all be called "Economy". If you search on most of the other OTA's, you can sometimes find the fare class during checkout or even as part of your search results, but you can't filter on it (Hipmunk is one that does support some of ITA's syntax for these filters, but not all). The buckets aren't always strictly more/less expensive, but they're usually not exposed very easily, if at all[2]. So, you're often left crawling from listing to listing, expanding to see if they are going to get you any miles. (I'll save the debate of whether miles are worth all the effort for another day.)

On ITA, it's not unreasonable to construct a query that says "During the month of November, show me round trips that are between 12 and 19 days that are going from Denver to either Narita or Haneda Airports, which will earn me more than 50% miles on either Delta or American or JAL, but also only ones that connect in Portland or Los Angeles, with no prop planes or overnight stops, and no <50 minute connections or 3+ hour connections". (I wouldn't actually specify all of these stipulations, but they're good for the example! :) )

[1] https://www.alaskaair.com/content/mileage-plan/how-to-earn-m...

[2] Delta, to its credit, does allow you to search by minimum fair class on its advanced search)

[will_hughes]

Using ITA Matrix is pretty easy. Using ITA Matrix to actually find better deals takes a lot of knowledge about how the airline systems work.

If you're only looking at flights out of a particular airport and with a specific (default) routing, you'll probably not find anything better.

If you know that certain airports are hubs, or that a particular carrier has a slightly longer route that takes you via a certain city and maybe have a longer layover then you might be able to find some really good deals.

Being in Australia, I really don't get to take advantage of these things at all.

[chx]

1. there's minconnect -- I can structure flights so I have time to sleep while in transit. You will never find those in a normal search engine, they try to achieve minimal time. There's also padconnect, if you don't like booking right on MCT -- which is often wise! -- then, as the name suggests, this will add N minutes to the prescribed minimal connection time.

2. As recently revealed by the company http://www.flyertalk.com/forum/27265924-post483.html you can avoid connections in certain countries which can be beneficial for visa purposes or certain personal privacy requirements ( I know a German physics professor who refuses to go to the US because of the fingerprinting. )

[enobrev]

ITA Matrix Powers Google Flights and Kayak, or at least parts therein.

[anewhnaccount]

Last time I tried I found the fact I was able to select multiple departure airports and multiple destinations and see a calendar of fares a great way to find the cheapest fares.

[CPLX]

A strong second to that, literally my favorite web tool in existence.

[dudurocha]

Thanks for Crash Bandicoot.

How did you go from games to developing airlines reservation and fares software?

[dmbaggett]

I knew two awesome people through the MIT AI Lab, one of whom co-founded Naughty Dog, and the other of whom co-founded ITA Software. :)

[ekianjo]

Awesome connections!

[auvi]

If you don't mind sharing, does ITA/Google Flights still use Common Lisp for the backend?

[breakds]

Yes. The core of the backend is still in Common Lisp (and C++). Many ITA hackers are still contributing to the Common Lisp community :)

[phillco]

Interesting -- didn't Google use to be super strict on the languages/runtimes they use in production? Did they make an exception for ITA?

[dmbaggett]

I don't actually know. But porting it would be a huge undertaking.

[jsnell]

It does.

[bluedino]

30M LoC sounds insane - what is the main reason behind there being so much code?

[dmbaggett]

It requires a lot of code because the rules that govern the use of fares on a ticket are insanely complicated -- literally beyond anything you could possibly imagine a priori. There are dozens of rule "categories", each larded up with tons of complexity. There's even "Rule Category 25: Fare By Rule" which is essentially a complete macro facility for fares, allowing carriers to literally take an arbitrary fare's rules, modify them in all kinds of random ways, and publish the result as a new "virtual" fare.

As for the GDSes, their code is primarily TPF assembly. So now add the LoC blow-up of using a very low-level language.

There are like 25 people in the world who understand this stuff, and half of them likely work at ITA/Google.

[akavel]

Also people at Sabre (worked there) and I presume Amadeus (those are two main players - surprised you seemed to "forget" to mention them?).

As to the "categories", people at Sabre working with them used to say in some cases you can not even be sure if the calculation (of a ticket price - "fare") will end in finite time, so the "macro language" is apparently Turing-complete!

[dmbaggett]

I wasn't meaning to slight either Amadeus or Sabre, and indeed they have most of the other half of the people who understand this stuff between them. The others are at various airlines around the world and at ATPCO.

I've written elsewhere that I personally think the original Sabre system was/is one of the most impressive accomplishments in the history of computing. Using modern tools made things easier for us at ITA, though we compensated for that by trying to compute the entire (very large) solution space for every query, where prior systems used heuristics.

[boggydepot]

Just some random question. Can you give some insight on how "Greenspuns Tenth Rule Of Programming" apply in this 30million LoC? thanks in advance

[boxfire]

https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule

[git_rancher]

Does all that complexity limit the amount of price competition airlines engage in?

[babuskov]

Java boilerplate? Just kidding. :)

[peterwwillis]

What are your favorite travel hacks?

[dmbaggett]

Never set foot on an airplane unless I absolutely have to. :)

[chromeextdev]

Why is that exactly?

[ethbro]

I'd imagine after staring at all the convoluted fare rules for a substantial amount of work time, anything to do with air travel becomes far less magical.

[brettproctor]

Do you know how expert flyer works under the hood? Always been curious about their credit model. Are they charging per query since they're paying per query to the GDS companies?

[EFVoice]

Hi Brett, we charge a monthly or a yearly subscription fee that at the Premium level includes unlimited searching. Details here: http://www.expertflyer.com/frequent-flyer-programs Email us if you want to discuss more at customercare@

[spelunker]

Seconded on ITA Matrix, great piece of software!

[NikolaeVarius]

There is a GDS that all (or most) vendors send data to

https://en.wikipedia.org/wiki/Global_Distribution_System

Downstream companies then query this data and show them to you. The competition comes from the complex caching and delivery mechanisms that the companies use to get you the best price.

It's not really an area that is ripe for disruption. The margins are extremely low and there are only really two companies that make a majority of the money. Expedia and the Priceline group.

[baby]

I think Amadeus is a popular one, matrix itasoftware is another one.

I know that for airlines like easyjet you need to visit their website to get fares though. (Or Google would have to build a site crawler)

[wodenokoto]

Is momondo under one of those? At least where I live it is the biggest by far and I always thought it was globally in the top.

[squeaky-clean]

> Does each airline have their own way of exporting this data? Is there a single entity that aggregates from all of them?

I work with a lot of smaller airlines, and from my experience I can tell you no. Many of them can't even tell you their actual route list or flight schedule or pricing logic. They typically buy a booking engine solution from just a few major players (Sabre, Amadeus, TravelPort) who do have centralized data, but access to them is slow, expensive (at scale) and restrictive.

I've been told by one of the GDS's that our use case doesn't apply for direct GDS access, and we have to go through the airlines' access method. I don't know if these booking engine solutions include access to an API or not, however most airlines have said either they don't have an available API, or don't know how to share access to one.

Some of the larger airlines do seem to have their stuff together. But we're just starting to work with them, and because of all the bureaucracy involved it will likely be months before anything happens.

The screen scraping method djhworld mentions is a worthwhile approach if you'd personally like to track a few routes. I've done the same to watch prices to NYC (I visit friends often) on a few of the low-cost airlines. It's nice because you can tailor the logic to your preferences. Such as don't bother searching flights that don't include a weekend in the trip; or set my alert threshold 25% higher when the trip involves a holiday or long weekend.

[geverett]

We built Hitlist (hitlistapp.com) to make this accessible to laypeople - I used to do the same thing you mention, scraping specific routes I knew I was going to fly. It's far from perfect but makes it easier to browse lots of date/destination options at a time.

[eddmc]

What is your specific use case? I used to work managing API's for one of the GDS companies

For screen scraping, there are a number of companies that do this already for multiple carriers. I'd suggest making use of one of these rather than reinventing the wheel. Example: http://xmldocs.travelfusion.com

[squeaky-clean]

My project definitely isn't applicable because it requires caching prices, I don't remember why we were declined access for our other teams/products though. I would have to ask them, as it's been a while.

About screen scraping, I guess it depends on how quick you are at writing scraping scripts (and if you find it fun). I could probably write one faster than I could register for that API. You're also adding a layer between you and the prices. You don't know how long Travelfusion has cached these prices for, while it's simple to write a script that opens the real booking engine to get correct prices. I typically only run mine when I'm ready to purchase a ticket, but don't know which dates, rather than running it constantly to find deals.

And if I were doing anything more than a few routes on a few carriers, I definitely would use a real API instead of reinventing the wheel. In this case I just think it's kinda fun.

[sluggg]

Your information is spot on. I have been working with Sabre daily for a few years now, and I don't want to think about the time I have wasted waiting for a Sabre response... it does go down once a week, which I guess is a perk

[melvinmt]

You should first start reading these great slides from Carl de Macken (ITA Software co-founder, the thing that now powers Google Flights) http://www.demarcken.org/carl/papers/ITA-software-travel-com...

…and then decide if you still want to pursue this.

[dmbaggett]

Short answer: you don't!

[jaza]

Great read! Incredibly thorough modelling of all components of the flight ticketing problem space, and of the computational complexities involved. Sets a high bar as an example of how to thoroughly model any search-related problem.

[skybrian]

In Google's case it comes from an acquisition [1]. Not all airlines share data. For example, you won't find Southwest fares except on their own website.

[1] http://www.itasoftware.com/about/

[toddkazakov]

> For example, you won't find Southwest fares except on their own website.

Wrong example. I have worked on the Southwest Project when I was at Amadeus.

[clueless123]

Southwest has it's own system, but does share their data through Sabre.. Fun Fact.. upto 2005 they still sent us the data hand delivered on a 9 track tape!

[djhworld]

I've known "regular" people in the past that have written small scale services for themselves that scrape prices daily from an airlines own booking system and then provide a service so you can track it and see if there are price fluctuations.

Not sure on the legality of it though.

[joaonunesk]

It is very expensive to get our hands on the technology components that power the core of such applications. [1]

If an airline sells seats, bags, and other ancillary services, why aren't they trying to find more sales channels, to sell more?

I believe that airlines haven't been pushing for innovation, and rather, have been slowed down by their very own current third party providers, or "core business partners".

Travelling technology applications or reservation systems, first appeared a long time ago [2] and because at that time, they were innovations, some airlines signed up for them, to make more sales by globally or regionally distributing all of their flight related information (seat or ancillary availability, fares, code share agreements, etc.). For this reason, we can consider reservation systems as the very first technological airline business partners.

Typically, the reservation systems provided innovation on a distribution factor, which is the travel agency distribution factor. Travel agencies could, and have been connecting to reservation systems and making purchases that increase the number of airline sales, resulting in a win-win, for both agencies and airlines. [3] Reservation systems have other goals, such as flight code share agreements, but these are out of scope. [5]

Today, these third party business organisations do not want to lose market share, so they make things expensive or maintain the old prices, and open very little doors on the borders of their systems.

Therefore, only typical rich individuals or companies can afford to purchase and maintain the costs of direct connections to these providers, which is a win-loss, win for the reservation systems, and loss for airlines.

Fortunately, companies such as Google (with QPX), perform a re-engineering of what these big reservation systems players do, and make it much more affordable and simple to implement [4], and with that, not only they improve the understanding effort of the whole business, but also, provide more sales leads to the airlines, and better user and developing experiences.

The future should be standardised, airlines should distant themselves from the reservation systems (if these do not innovate), and open the doors of their businesses, by creating booking process APIs, that allow, developers and startups, to create powerful and innovative applications, that bring sales.

[1] https://www.tnooz.com/article/why-is-the-travel-industry-so-... [2] https://en.wikipedia.org/wiki/Sabre_(computer_system) [3] https://en.wikipedia.org/wiki/Travel_agency [4] https://developers.google.com/qpx-express/v1/pricing [5] https://en.wikipedia.org/wiki/Codeshare_agreement