[nicolasd]

Serious question: How would you suggest to write a test that prevents this? You wouldn't make an assertion on clearTextPassword !== passwordHint. While developing I would think "Who will ever do that? That would be insane".

But yes, even if there is no test, this should have been caught in code review or latest when testing the OS.

[eru]

Yes, with 'example based testing' this is hard to come up with. With property based testing, it's not so hard to test this kind of UI things:

You test a UI by basically throwing sequences of interactions at it. Some of the properties you'd want to assert:

* Given two interaction sequences that only differ in what they do to the password hint field in the UI, the result should only differ differ in the returned password hint. (Or alternatively neither should finish the UI dialogue.)

* As a dual: two interaction sequences that share the same interaction with the password hint field should yield the same password hint field. (In this case it is acceptable for them to differ in whether they actually finish the dialogue.)

Those two are fairly generic, so you can imagine setting that up as a general framework for all your data input fields in your UIs. It should work backwards as well, eg to assert that eg the UI should look the same no matter what password (/ password hash) is stored, to make sure you are not leaking any information.

See eg https://fsharpforfunandprofit.com/posts/property-based-testi... for more background, and http://hypothesis.works/articles/incremental-property-based-... for a real world example.

As for 'should have been caught in code review': yes, but humans are fallible and they should get all the help we can give them. To see for yourself, have a look at the example (a simple runlength encoding) at the top of http://hypothesis.readthedocs.io/en/latest/quickstart.html and see whether you can spot the obvious error just by reviewing the code.

[nicolasd]

Thanks for the detailed answer and coming up with a property-based test that is not a "1+1=2" example :)

[eru]

I think I might have stumbled upon a new category of common properties here. (At least new to me.) Basically, test that some subset of your output variables is a function of a specific subset of your input variables. And conversely, test that some subset of your output variables is independent (under some conditions) of some subset of your input variables.

My go-to example for property based testing isn't addition, but idempotence. Idempotence is a concept well known from REST apis to the general programming public, and also in sorting or clean up data. And it's easy to state in code, eg: sorted(sorted(x)) == sorted(x).

[bsaul]

It seems like the error was in the creation of an intermediate data structure ( minimal dictionary something) in a storageKit framework. I’m not talking about the disk util app ( which doesn’t contain any bug in itself apparently)

To give a better answer : I’m not knowledgeable of the whole source for the framework, but there are only two cases :

- either the outputs of your critical function are testable and then just test it.

- or they’re not ( most often because of side effects), in which case you should extract the critical part in purer function, and test that.

It’s a great benefit of unit test : they force you to isolate side effects from business logic, to be able to test the latter.

In that case, if the function isn’t testable, then maybe the creation of this intermediate minimal dictionary should belong in its own function that just does the data mapping.

[abalone]

You wouldn’t need to test it like that. What’s missing here is simply a test of that StorageKit API that the GUI uses. The test would be straightforward: set all the options and see if the resulting volume is created correctly. So for the hint your test would generate a value — a good test will exercise a range of values over multiple iterations — and verify the hint on the created volume.

No doubt this basic test was done on the command line API, which works correctly. It’s likely the problem here is they created a separate StorageKit API just for Disk Utility to use and got sloppy with the unit test for that. A good example of why it is a good idea to try to have GUIs and command lines use one code path whenever possible.

[alextooter]

I have developed a UI test automation software years ago, it should be getHintFromUI() == passwordHint.Here getHintFromUI is a function which get string from User Interface directly.

[maccard]

I noticed when I upgraded that they block you from setting your account password to the same as your iCloud password, so they've obviously considered that people will do stupid things.

[ShinTakuya]

How about a simple test giving the function a set of parameters and then asserting that the output dictionary (besides the password hash) equals the expected output? That would have caught this case with ease.