[femto113]

I have a confusion about ID structure/format:

   The ID is composed of two parts: a millisecond time and a
   sequence number.  The number after the dot is the
   sequence number, and is used in order to distinguish
   entries added in the same millisecond. 

Does this mean for example that 1506872463535.11 comes after 1506872463535.2 (because 11 > 2)? If so that means treating these as decimals (which will be easy to do inadvertently) will yield the wrong order (as would sorting them lexicographically). If so it seems like something other than a decimal point would be a better separator (colon perhaps).

[antirez]

Yes actually maybe it's a good idea to change the point with something else. Thanks for the hint.

[ryanschneider]

Supporting `MAXSIZE` as well as `MAXLEN` on `XADD` would also handle a nice Kafka feature (the ability to define your log size in either number of messages or size on disk).

Something like: `XADD MAXSIZE ~ 2147484000 * foo bar` to cap the stream at 2GB + 1 node.

[dvirsky]

And if ids are timestamps, maybe we can define it as MAXTIME as well.

[agumonkey]

What were or would be your prefered alternative syntax ? `:` ?

[antirez]

Not sure... : looks ok actually, even _ or - could make some sense. The # is a bit too heavy on the eyes :-)

[femto113]

I'll probably eventually hate myself for even bringing this up but I can't help but notice the similarity between this ID structure and Version 1 (aka timestamp) UUIDs. While I wouldn't go as far as recommending that you fully adopt that form, it might be worth considering if you could make these IDs compatible with UUIDs by defining a canonical transform. The critical differences are:

- UUIDs use a different epoch (15 Oct 1582 vs 1 Jan 1970) - UUIDs count 100 ns blocks instead of ms - UUIDs include a 6 byte "node id" - UUIDs allow only up to 15 bits of "sequence"

I think that last one is the biggest deal, since as currently specced redis allows 64 bits of sequence, which is obviously much bigger than 15. The options I see are either up the time resolution used by redis, encode some of redis's sequence bits into the UUID's time bits, or just live with it as a limitation--in practice 2^15 is a lot of messages to get in a single millisecond (though in cases of clocks jumping back might not be too much).

You'd also need to come up with some thing for the node id, perhaps the first 6 bytes of a cluster node ID or similar.

[femto113]

Thinking about this some more I realized you could also encode sequence into the low order bits of the timestamp, and rereading the RFC showed it actually makes this recommendation[1]. There are 10000 100-nanosecond periods per millisecond which gives about 13 more bits. Between that and the 13-15 bits available in the clock sequence you've got 26+ bits of sequence or ~67MM values per millisecond.

Since 64 bits is overkill for milliseconds (45 bits covers the next 1000 years or so) I was thinking you could put 2 bytes of the node id in the high order bytes there (perhaps could call this the "clock id"?) and the remaining 4 bytes of the node id could go in the high order bytes of the sequence, which would still leave 32 bits for actual sequence values (but we should only use 26 or so). This means we'd get a translation roughly as follows (numbering bytes and bits from high to low significance):

   Redis                                   Version 1 UUID
      Timestamp
        Byte 0-1  "clock id"               Bytes 4&5 of node id
        Byte 2-7  millis since 1 Jan 1970  * 10000 => ~45 high order timestamp bits
      Sequence
        Byte 0-3  "node id"                Bytes 0-3 of node id
        Byte 4-7
           6 bits wasted space             ignored
          26 bits actual sequence value
            13 high order bits             => clock sequence
            13 low order bits              => low order timestamp bits

Another implication of this scheme is that if redis has access to a clock that offers higher than millisecond resolution it could store everything more precise than millisecond into the sequence portion of the id.

On a side note it seems that the clock sequence in the UUID is intended to be reset to a random value at start up and every time a clock jump is detected rather than just incremented. Redis could do something similar by incrementing some of the 13 high-order bits of the sequence every time a clock jump is detected (and/or if the 13 low-order bits overflow)

[1] https://tools.ietf.org/html/rfc4122#section-4.2.1.2

[lsiebert]

I think a mostly vertical symbol is better, ":" or "|" is preferably to "_" or "-".

[simonhamp]

I agree. Perhaps a forward-slash (/) to denote subordination, e.g. 1507035873/11

[anyfoo]

Why is that?

[agumonkey]

verticality convey difference in kind of information better IMO

[rakoo]

Why keep the separation at all ? Are clients expected to be able to query for a given timestamp precisely ? Because then you get all the problems with clock synchronization, especially given that the Streams' clock is monotonic and I'd expect clients' clock to not be

[unkown-unknowns]

It might be useful to be able to query by server time regardless of whether your client clock is in sync. You retrieve some set of data and the next time you can ask the server to give you everything newer than x, where x was the highest time stamp you got from the server previously.

[rakoo]

Yes exactly, you want to ask what is newer than x, where x is the last event you're aware of, but you don't really care about the date and time in that case. If you just store the last id given by redis Streams naively then you don't even care that they're timestamps; at that point my question is, why even bother with the distinction. Just ask for everything after x and be done with it.

[lsiebert]

Redis also has TIME to get the current server time with milliseconds and the unix time stamp. I'm reasonably sure that's what's being used to get the first part of the ID anyway.

[rakoo]

Ah, I didn't know that. Although the post says that the timestamp of an id might also be the timestamp of the last message, since the clock can go backwards, so in the worst case a client might get some duplicate messages.

[unkown-unknowns]

I vote that you use a dash instead.

[zedpm]

It turns out that's what it will be [0], as dash retains the ability to easily copy the whole identifier in most terminals.

[0]: https://github.com/antirez/redis/commit/1189d90d749c84e98424...

[pookeh]

You can also have a look at the technique used here to create collision-free sequential unique IDs across a cluster, even if it is just for inspiration: https://www.npmjs.com/package/cuid

Example:

c - h72gsb32 - 0000 - udoc - l363eofy

The groups, in order, are:

1. 'c' - identifies this as a cuid, and allows you to use it in html entity ids. The fixed value helps keep the ids sequential.

2. Timestamp

3. Counter - a single process might generate the same random string. The weaker the pseudo-random source, the higher the probability. That problem gets worse as processors get faster. The counter will roll over if the value gets too big.

4. Client fingerprint. For example, the first two chars are extracted from the process.pid. The next two chars are extracted from the hostname.

5. Pseudo random (Math.random())

[ecesena]

I don't think you can ever think to a total order between events. In your example. My understanding is that the 2 events in your example happened in parallel (by redis definition of time granularity), and there's no correct ordering between the two. What I want to say is that even if you change to a ":" you'd get wrong results.

I think antirez is saying that, by serializing the id in this way, you can also get timeseries at the ms precision for free.

Edit: nvm, antirez just replied :)

[archit3cture]

In the XADD example, you have

XADD mystream * sensor-id 1234 temperature 10.5

to let the server choose an id.

It could also be interesting to have

XADD mystream 1506872463535.* sensor-id 1234 temperature 10.5

to be able to add an element in a specific msec bucket

[ihattendorf]

What about regions that treat the comma as a decimal separator? I agree with the other commenter that this is no different than using periods in IPv4 addresses.

[IanCal]

> What about regions that treat the comma as a decimal separator?

A good reason not to change it from a period to a comma, not really relevant to whether to change it from a period to something else.

> I agree with the other commenter that this is no different than using periods in IPv4 addresses.

IP addresses have no useful concept of "before" or "after", whereas these do.

[anyfoo]

As a member of such a region, I suspect that those regions a well aware that the prevalent notation throughout programming uses '.' as a decimal separator. IPv4 addresses usually have 4 components and do not have an inherently fractional unit as their first component.

[mijamo]

Disagree. '.' is used for version numbers everywhere, even when it's only 2 components (ex: Wordpress versioning, Django etc.) and it seems pretty clear that 1.11 > 1.2 there.

Once the specification is stated clearly as it is right now it is not a problem.

Another symbol could be used (for instance #) but I really don't see a need for that.

[sterex]

Perhaps decoupling the <$.#> as (# x 0.01) would solve the problem right?

Not sure if that's what they do, but it isn't very complex.

[twic]

The dot doesn't make that a decimal, any more than it makes IP addresses or version numbers decimals.

As for treating them as decimals inadvertently, well, hopefully client libraries will expose IDs as pairs of integers, not as strings. If users convert them into strings and then back into meaningless pseudo-decimals, well, great, we'll have an entertaining post about someone's outage to read.

[hueving]

This is a terrible attitude to have when responding to an obvious potential UX confusion, particularly when it will only come up in edge cases (>10 per millisecond).

[maerF0x0]

10 per milli as an edge case is domain specific.

[rjeli]

But, it has milliseconds on the left. Before I read the explanation, I immediately thought it was a floating point timestamp. IP addresses have no reasonable decimal interpretation.

[anyfoo]

It's not much of a problem for (v4) IPs, because they almost always consist of 4 numbers separated by a dot, making them immediately distinguishable from decimal numbers. If two-component IPs were common (they are sometimes seen in CIDR notation, but not often), the dot would have been an unfortunate separator choice as well.

For versions with only two components, I would argue that the dot can be confusing already.

Why use a separator that has the potential of confusion when there are several other choices with less potential?

[nurettin]

I had the same thoughts until the haha-outage hyperbole. Perhaps this little feature spurring so much discussion about delimiters is caused by a lack of thought by the developer, releasing an idea before it fully matured. A sort of race towards innovation mixed with a hint of it-works-ship-it.

[xapata]

Ah yes, like trap answers on a multiple choice exam. I suppose the Zen of that design would be: "There should be more than one obvious way to do it, but only one correct way."